Package Relay Proposal

Package Relay Proposal
2022-05-17 17:56 ` Greg Sanders
                   ` (2 more replies)
  
From: Gloria Zhao @ 2022-05-17 16:01 UTC (permalink / raw)
  To: Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 27392 bytes --]

Hi everybody,

I’m writing to propose a set of p2p protocol changes to enable package
relay, soliciting feedback on the design and approach. Here is a link
to the most up-to-date proposal:

https://github.com/bitcoin/bips/pull/1324

If you have concept or approach feedback, *please respond on the
mailing list* to allow everybody to view and participate in the
discussion. If you find a typo or inaccurate wording, please feel free
to leave suggestions on the PR.

I’m also working on an implementation for Bitcoin Core.


The rest of this post will include the same contents as the proposal,
with a bit of reordering and additional context. If you are not 100%
up-to-date on package relay and find the proposal hard to follow, I
hope you find this format more informative and persuasive.


==Background and Motivation==

Users may create and broadcast transactions that depend upon, i.e.
spend outputs of, unconfirmed transactions. A “package” is the
widely-used term for a group of transactions representable by a
connected Directed Acyclic Graph (where a directed edge exists between
a transaction that spends the output of another transaction).

Incentive-compatible mempool and miner policies help create a fair,
fee-based market for block space. While miners maximize transaction
fees in order to earn higher block rewards, non-mining users
participating in transaction relay reap many benefits from employing
policies that result in a mempool with the same contents, including
faster compact block relay and more accurate fee estimation.
Additionally, users may take advantage of mempool and miner policy to
bump the priority of their transactions by attaching high-fee
descendants (Child Pays for Parent or CPFP).  Only considering
transactions one at a time for submission to the mempool creates a
limitation in the node's ability to determine which transactions have
the highest feerates, since it cannot take into account descendants
until all the transactions are in the mempool. Similarly, it cannot
use a transaction's descendants when considering which of two
conflicting transactions to keep (Replace by Fee or RBF).

When a user's transaction does not meet a mempool's minimum feerate
and they cannot create a replacement transaction directly, their
transaction will simply be rejected by this mempool. They also cannot
attach a descendant to pay for replacing a conflicting transaction.
This limitation harms users' ability to fee-bump their transactions.
Further, it presents a security issue in contracting protocols which
rely on **presigned**, time-sensitive transactions to prevent cheating
(HTLC-Timeout in LN Penalty [1] [2] [3], Unvault Cancel in Revault
[4], Refund Transaction in Discreet Log Contracts [5], Updates in
eltoo [6]). In other words, a key security assumption of many
contracting protocols is that all parties can propagate and confirm
transactions in a timely manner.

In the past few years, increasing attention [0][1][2][3][6] has been
brought to **pinning attacks**, a type of censorship in which the
attacker uses mempool policy restrictions to prevent a transaction
from being relayed or getting mined.  TLDR: revocation transactions
must meet a certain confirmation target to be effective, but their
feerates are negotiated well ahead of broadcast time. If the
forecasted feerate was too low and no fee-bumping options are
available, attackers can steal money from their counterparties. I walk
through a concrete example for stealing Lightning HTLC outputs at
~23:58 in this talk [7][8].  Note that most attacks are only possible
when the market for blockspace at broadcast time  demands much higher
feerates than originally anticipated at signing time. Always
overestimating fees may sidestep this issue temporarily (while mempool
traffic is low and predictable), but this solution is not foolproof
and wastes users' money. The feerate market can change due to sudden
spikes in traffic (e.g. huge 12sat/vB dump a few days ago [9]) or
sustained, high volume of Bitcoin payments (e.g.  April 2021 and
December 2017).

The best solution is to enable nodes to consider packages of
transactions as a unit, e.g. one or more low-fee parent transactions
with a high-fee child, instead of separately. A package-aware mempool
policy can help determine if it would actually be economically
rational to accept a transaction to the mempool if it doesn't meet fee
requirements individually. Network-wide adoption of these policies
would create a more purely-feerate-based market for block space and
allow contracting protocols to adjust fees (and therefore mining
priority) at broadcast time.  Some support for packages has existed in
Bitcoin Core for years. Since v0.13, Bitcoin Core has used ancestor
packages instead of individual transactions to evaluate the incentive
compatibility of transactions in the mempool [10] and select them for
inclusion in blocks [11].

Package Relay, the concept of {announcing, requesting, downloading}
packages between nodes on the p2p network, has also been discussed for
many years. The earliest public mention I can find is from 2015 [12].
The two most common use cases for package relay are fee-bumping
otherwise-too-low-fee transactions and reducing the amount of orphans.
It seems uncontroversial to say that everybody desires package relay
conceptually, with varying degrees of urgency. Lots of work has been
done by others over the past few years, from which I've taken
inspiration from [13][14][15][16].

My approach has been to split the project into two components: (1) Package
Mempool Accept, which includes validation logic and mempool policy.
(3) Package Relay, which includes the p2p protocol changes.

Progress so far:
After discussions with various developers of contracting protocols
(with heavier emphasis towards LN), it was determined that a
package containing a child with all of its unconfirmed parents
(child-with-unconfirmed-parents or 1-child-multi-parent package) would
be sufficient for their use case, i.e. fee-bumping presigned
transactions. A child-with-unconfirmed-parents package has several
properties that make many things easier to reason about.

A few months ago, I proposed a set of policies for safe package
validation and fee assessment for packages of this restricted
topology [17]. A series of PRs implementing this proposal have
been merged into Bitcoin Core [18].

Theoretically, developing a safe and incentive-compatible package
mempool acceptance policy is sufficient to solve this issue. Nodes
could opportunistically accept packages (e.g. by trying combinations
of transactions rejected from their mempools), but this practice would
likely be inefficient at best and open new Denial of Service attacks
at worst. Additional p2p messages may enable nodes to request and
share package validation-related information with one another in a
more communication-efficient way.

Given that only package RBF remains for package mempool accept, and we
can make progress on p2p and mempool in parallel, I think it’s
appropriate to put forward a package relay proposal.

==Proposal==

This proposal contains 2 components: a “generic” package relay
protocol and an extension of it, child-with-unconfirmed-parents
packages, as version 1 package relay. Another version of packages,
“tx-with-unconfirmed-ancestors” can be created to extend package relay
for eliminating orphans.

===Generic Package Relay===

Two main ideas are introduced:

Download and validate packages of transactions together.

Provide information to help peers decide whether to request and/or how
to validate transactions which are part of a package.

====Intended Protocol Flow====

Due to the asynchronous nature of a distributed transaction relay
network, nodes may not receive all of the information needed to
validate a transaction at once. For example, after a node completes
Initial Block Download (IBD) and first starts participating in
transaction relay with an empty mempool, it is common to receive
orphans. In such scenarios where a node is aware that it is missing
information, a ''receiver-initiated'' dialogue is appropriate:

1. Receiver requests package information.

2. The sender provides package information, including the wtxids of
   the transactions in the package and anything else that might be
relevant (e.g. total fees and size).

3. The reciever uses the package information to decide how to request
   and validate the transactions.

Sometimes, no matter what order transactions are received by a node,
validating them individually is insufficient. When the sender is aware
of additional information that the receiver needs to accept a package,
a proactive ''sender-initiated'' dialogue should be enabled:

1. Sender announces they have package information pertaining to a
   transaction that might otherwise be undesired on its own.

2. The receiver requests package information.

3. The sender provides package information, including the wtxids of
   the transactions in the package and anything else that might be
relevant (e.g. total fees and size).

4. The reciever uses the package information to decide how to request
   and validate the transactions.

Package relay is negotiated between two peers during the version
handshake. Package relay requires both peers to support wtxid-based
relay because package transactions are referenced by their wtxid.

====New Messages====

Three new protocol messages are added for use in any version of
package relay. Additionally, each version of package relay must define
its own inv type and "pckginfo" message version, referred to in this
document as "MSG_PCKG" and "pckginfo" respectively. See
BIP-v1-packages for a concrete example.

=====sendpackages=====

{|
|  Field Name  ||  Type  ||  Size  ||  Purpose
|-
|version || uint32_t || 4 || Denotes a package version supported by the
node.
|-
|max_count || uint32_t || 4 ||Specifies the maximum number of transactions
per package this node is
willing to accept.
|-
|max_weight || uint32_t || 4 ||Specifies the maximum total weight per
package this node is willing
to accept.
|-
|}

1. The "sendpackages" message has the structure defined above, with
   pchCommand == "sendpackages".

2. During version handshake, nodes should send a "sendpackages"
   message indicate they support package relay and may request
packages.

3. The message should contain a version supported by the node. Nodes
   should send a "sendpackages" message for each version they support.

4. The "sendpackages" message MUST be sent before sending a "verack"
   message. If a "sendpackages" message is received afer "verack", the
sender should be disconnected.

5. If 'fRelay==false' in a peer's version message, the node must not
   send "sendpackages" to them. If a "sendpackages" message is
received by a peer after sending `fRelay==false` in their version
message, the sender should be disconnected.

6.. Upon receipt of a "sendpackages" message with a version that is
not supported, a node must treat the peer as if it never received the
message.

7. If both peers send "wtxidrelay" and "sendpackages" with the same
   version, the peers should announce, request, and send package
information to each other.

=====getpckgtxns=====

{|
|  Field Name  ||  Type  ||  Size  ||   Purpose
|-
|txns_length||CompactSize||1 or 3 bytes|| The number of transactions
requested.
|-
|txns||List of wtxids||txns_length * 32|| The wtxids of each transaction in
the package.
|}

1. The "getpckgtxns" message has the structure defined above, with
   pchCommand == "getpckgtxns".

2. A "getpckgtxns" message should be used to request all or some of
   the transactions previously announced in a "pckginfo" message,
specified by witness transactiosome id.

3. Upon receipt of a "getpckgtxns" message, a node must respond with
   either a "pckgtxns" containing the requested transactions or a
"notfound" message indicating one or more of the transactions is
unavailable. This allows the receiver to avoid downloading and storing
transactions that cannot be validated immediately.

4. A "getpckgtxns" message should only be sent if both peers agreed to
   send packages in the version handshake. If a "getpckgtxns" message
is received from a peer with which package relay was not negotiated,
the sender should be disconnected.

=====pckgtxns=====

{|
|  Field Name  ||  Type  ||  Size  ||   Purpose
|-
|txns_length||CompactSize||1 or 3 bytes|| The number of transactions
provided.
|-
|txns||List of transactions||variable|| The transactions in the package.
|}

1. The "pckgtxns" message has the structure defined above, with
   pchCommand == "pckgtxns".

2. A "pckgtxns" message should contain the transaction data requested
   using "getpckgtxns".

3. A "pckgtxns" message should only be sent to a peer that requested
   the package using "getpckgtxns". If a node receives an unsolicited
package, the sender should be disconnected.

4. A "pckgtxns" message should only be sent if both peers agreed to
   send packages in the version handshake. If a "pckgtxns" message is
received from a peer with which package relay was not negotiated, the
sender should be disconnected.

===Version 1 Packages: child-with-unconfirmed-parents===

This extends package relay for packages consisting of one transaction
and all of its unconfirmed parents,by defining version 1 packages, a
pckginfo1 message, and a MSG_PCKG1 inv type. It enables the use case
in which a child pays for its otherwise-too-low-fee parents and their
mempool conflict(s).

====Intended Protocol Flow====

When relaying a package of low-fee parent(s) and high-fee child, the
sender and receiver do the following:

1. Sender announces they have a child-with-unconfirmed-parents package
   for a child that pays for otherwise-too-low-fee parent(s) using
"inv(MSG_PCKG1)".

2. The receiver requests package information using
   "getdata(MSG_PCKG1)".

3. The sender provides package information using "pckginfo1",
   including the blockhash of the sender's best block, the wtxids of
the transactions in the package, their total fees and total weight.

4. The reciever uses the package information to decide how to request
   the transactions. For example, if the receiver already has some of
the transactions in their mempool, they only request the missing ones.
They could also decide not to request the package at all based on the
fee information provided.

5. Upon receiving a "pckgtxns", the receiver submits the transactions
   together as a package.

====New Messages====

A new inv type, "MSG_PCKG1", and new protocol message, "PCKGINFO1",
are added.

=====pckginfo1=====

{|
|  Field Name  ||  Type  ||  Size  ||   Purpose
|-
|blockhash || uint256 || 32 || The chain tip at which this package is
defined.
|-
|pckg_fee||CAmount||4|| The sum total fees paid by all transactions in the
package.
|-
|pckg_weight||int64_t||8|| The sum total weight of all transactions in the
package.
|-
|txns_length||CompactSize||1 or 3 bytes|| The number of transactions
provided.
|-
|txns||List of wtxids||txns_length * 32|| The wtxids of each transaction in
the package.
|}


1. The "pckginfo1" message has the structure defined above, with
   pchCommand == "pckginfo1".

2. A "pckginfo1" message contains information about a version 1
   package (defined below), referenced by the wtxid of the transaction
it pertains to and the current blockhash.

3. Upon receipt of a "pckginfo1" message, the node should decide if it
   wants to validate the package, request transaction data if
necessary, etc.

4. Upon receipt of a malformed "pckginfo1" message or package that
   does not abide by the max_count, max_weight, or other rules
specified by the version agreed upon in the initial negotiation, the
sender should be disconnected.  If a node receives a "pckginfo1"
message for which the "pckg_fee" or "pckg_weight" do not reflect the
true total fees and weight, respectively, or the transactions in the
package, the message is malformed.

5. A node MUST NOT send a "pckginfo1" message that has not been
   requested by the recipient. Upon receipt of an unsolicited
"pckginfo1", a node should disconnect the sender.

6. A "pckginfo1" message should only be sent if both peers agreed to
   send version 1 packages in the version handshake. If a "pckginfo1"
message is received from a peer with which package relay was not
negotiated, the sender should be disconnected.

=====MSG_PCKG1=====

1. A new inv type (MSG_PCKG1 == 0x6) is added, for use in inv messages
   and getdata requests pertaining to version 1 packages.

2. As an inv type, it indicates that both transaction data and version
   1 package information are available for the transaction. The
transaction is referenced by its wtxid. As a getdata request type, it
indicates that the sender wants package information for the
transaction.

3. Upon receipt of a "getdata" request for "MSG_PCKG1", the node
   should respond with the version 1 package corresponding to the
requested transaction and its current chain tip, or with NOTFOUND.
The node should not assume that the sender is requesting the
transaction data as well.

====Child With Parent Packages Rules====

A child-with-unconfirmed-parents package sent between nodes must abide
by the rules below, otherwise the package is malformed and the sender
should be disconnected.

A version 1 or ''child-with-unconfirmed-parents'' package can be
defined for any transaction that spends unconfirmed inputs. The child
can be thought of as the "representative" of the package. This package
can be uniquely identified by the transaction's wtxid and the current
chain tip block hash.

A ''child-with-unconfirmed-parents'' package MUST be:

1. ''Sorted topologically.'' For every transaction t in the package,
   if any of t's parents are present in the package, the parent must
appear somewhere in the list before t. In other words, the
transactions must be sorted in ascending order of the number of
ancestors present in the package.

2. ''Only 1 child with unconfirmed parents.'' The package must consist
   of one transaction and its unconfirmed parents. There must not be
any other transactions in the package. Other dependency relationships
may exist within the package (e.g. one parent may spend the output of
another parent) provided that topological order is respected.

3. ''All unconfirmed parents.'' All of the child's unconfirmed parents
   must be present.

4. ''No conflicts.'' None of the transactions in the package may
   conflict with each other (i.e.  spend the same prevout).

5. ''Total fees and weight.'' The 'total_fee' and 'total_weight'
   fields must accurately represent the sum total of all transactions'
fees and weights as defined in BIP141, respectively.

Not all of the child's parents must be present; the child transaction
may also spend confirmed inputs. However, if the child has confirmed
parents, they must not be in the package.

While a child-with-unconfirmed-parents package is perhaps most
relevant when the child has a higher feerate than its parents, this
property is not required to construct a valid package.

====Clarifications====

''Q: Under what circumstances should a sender announce a
child-with-unconfirmed-parents package?''

A child-with-unconfirmed-parents package for a transaction should be
announced when it meets the peer's fee filter but one or more of its
parents don't; a "inv(MSG_PCKG1)" instead of "inv(WTX)" should be sent
for the child. Each of the parents which meet the peer's fee filter
should still be announced normally.

''Q: What if a new block arrives in between messages?''

A child-with-unconfirmed-parents package is defined for a transaction
based on the current chain state. As such, a new block extending the
tip may decrease the number of transactions in the package (i.e. if
any of the transaction's parents were included in the block). In a
reorg, the number of transactions in the package may decrease or
increase (i.e. if any of the transaction's parents were included in a
block in the previous chain but not the new one).

If the new block arrives before the "getdata" or "pckginfo1", nothing
needs to change.

If the new block arrives before "getpckgtxns" or before "pckgtxns",
the receiver may need to re-request package information if the block
contained a transaction in the package. If the block doesn't contain
any transactions in the package, whether it extends the previous tip
or causes a reorg, nothing needs to change.

''Q: Can "getpckgtxns" and "pckgtxns" messages contain only one
transaction?''

Yes.

===Further Protocol Extensions===

When introducing a new type of package, assign it a version number "n"
and use an additional "sendpackages" message during version handshake
to negotiate support for it. An additional package information message
"pckginfon" and inv type "MSG_PCKGn" should be defined for the type of
package.  However, "getpckgtxns" and "pckgtxns" do not need to be
changed.

Example proposal for tx-with-unconfirmed-ancestors package relay: [19]

===Compatibility===

Older clients remain fully compatible and interoperable after this
change. Clients implementing this protocol will only attempt to send
and request packages if agreed upon during the version handshake.

===Package Erlay===

Clients using BIP330 reconciliation-based transaction relay (Erlay)
are able to use package relay without interference. In fact, a package
of transactions may be announced using both Erlay and package relay.
After reconciliation, if the initiator would have announced a
transaction by wtxid but also has package information for it, they may
send "inv(MSG_PCKG)" instead of "inv(WTX)".

===Rationale===

====P2P Message Design====

These p2p messages are added for communication efficiency and, as
such, one should measure alternative solutions based on the resources
used to communicate (not necessarily trustworthy) information: We
would like to minimize network bandwidth, avoid downloading a
transaction more than once, avoid downloading transactions that are
eventually rejected, and minimize storage allocated for
not-yet-validated transactions.

Consider these (plausible) scenarios in transaction relay:

Alice (the "sender") is relaying transactions to Bob (the "receiver").
Alice's mempool has a minimum feerate of 1sat/vB and Bob's has a
minimum feerate of 3sat/vB. For simplicity, all transactions are
1600Wu in virtual size and 500 bytes in serialized size. Apart from
the spending relationships specified, all other inputs are from
confirmed UTXOs.

1. Package {A, B} where A pays 0 satoshis and B pays 8000 satoshis in
   fees.

2. Package {C, D} where C pays 0 satoshis and D pays 1200 satoshis in
   fees.

3. Package {E, F, G, H, J} that pays 4000, 8000, 0, 2000, and 4000
   satoshis in fees, respectively.

====Alternative Designs Considered====

''Package Information Only:'' Just having "pckginfo" gives enough
information for the receiver to accept the package. Omit the
"getpckgtxns" and "pckgtxns" messages. While this option is a good
fallback if batched transaction download fails for some reason, it
shouldn't be used as the default because it 'always' requires storage
of unvalidated transactions.

''No Package Information Round:'' Instead of having a package
information round, just use the child's wtxid to refer to the package
and always send the entire package together. This would cause nodes to
redownload duplicate transactions.

I have also created a slidedeck exploring various alternative designs
and some examples in which they fall flat [20]. Please feel free to
suggest other alternatives.

====Versioning System====

This protocol should be extensible to support multiple types of
packages based on future desired use cases. Two "flavors" of
versioning were considered:

1. When package mempool acceptance is upgraded to support more types
   of packages, increment the version number (similar to Erlay).
During version handshake, peers negotiate which version of package
relay they will use by each sending one "sendpackages" message.

2. When introducing another type of package, assign a version number
   to it and announce it as an additional supported version (similar
to Compact Block Relay). During version handshake, peers send one
"sendpackages" message for each version supported.

The second option was favored because it allows different parameters
for different versions.  For example, it should be possible to support
both "arbitrary topology but maximum 3-transaction" package as well as
"child-with-unconfirmed-parents with default mempool ancestor limits"
packages simultaneously.

==Acknowledgements==

I hope to have made it abundantly clear that this proposal isn’t
inventing the concept of package relay, and in fact builds upon years
of work by many others, including Suhas Daftuar and Antoine Riard.

Thank you to John Newbery and Martin Zumsande for input on the design.

Thank you to Matt Corallo, Christian Decker, David Harding, Antoine
Poinsot, Antoine Riard, Gregory Sanders, Chris Stewart, Bastien
Teinturier, and others for input on the desired interface for
contracting protocols.

Looking forward to hearing your thoughts!

Best,
Gloria

[0]:
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-January/019817.html
[1]:
https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-April/002639.html
[2]:
https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-June/002758.html
[3]: https://github.com/t-bast/lightning-docs/blob/master/pinning-attacks.md
[4]:
https://github.com/revault/practical-revault/blob/master/transactions.md#cancel_tx
[5]:
https://github.com/discreetlogcontracts/dlcspecs/blob/master/Transactions.md#refund-transaction
[6]: https://gist.github.com/instagibbs/60264606e181451e977e439a49f69fe1
[7]:
https://btctranscripts.com/adopting-bitcoin/2021/2021-11-16-gloria-zhao-transaction-relay-policy/#lightning-attacks
[8]: https://youtu.be/fbWSQvJjKFs?t=1438
[9]:
https://www.reddit.com/r/Bitcoin/comments/unew4e/looks_like_70_mvb_of_transactions_just_got_dumped/
[10]: https://github.com/bitcoin/bitcoin/pull/7594
[11]: https://github.com/bitcoin/bitcoin/pull/7600
[12]: https://github.com/bitcoin/bitcoin/pull/6455#issuecomment-122716820
[13]: https://gist.github.com/sdaftuar/8756699bfcad4d3806ba9f3396d4e66a
[14]: https://github.com/bitcoin/bitcoin/issues/14895
[15]: https://github.com/bitcoin/bitcoin/pull/16401
[16]: https://github.com/bitcoin/bitcoin/pull/19621
[17]:
https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-September/019464.html
[18]: https://github.com/users/glozow/projects/5/views/4?layout=board
[19]: https://gist.github.com/glozow/9b321cd3ef6505135c763112033ff2a7
[20]:
https://docs.google.com/presentation/d/1B__KlZO1VzxJGx-0DYChlWawaEmGJ9EGApEzrHqZpQc/edit?usp=sharing

[-- Attachment #2: Type: text/html, Size: 31833 bytes --]

 permalink raw reply	[|nested]
2022-05-17 16:01 Gloria Zhao

  2022-05-17 20:45   ` Gloria Zhao
  2022-05-18  0:35 ` Anthony Towns
  2022-06-17 20:08 ` Antoine Riard
  
From: Greg Sanders @ 2022-05-17 17:56 UTC (permalink / raw)
  To: Gloria Zhao, Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 29704 bytes --]

Hi Gloria,

Thanks for working on this important proposal!

Still a lot to digest, but I just had on area of comment/question:

> A child-with-unconfirmed-parents package sent between nodes must abide by
the rules below, otherwise the package is malformed and the sender should
be disconnected.

> However, if the child has confirmed parents, they must not be in the
package.

If my naive understanding is correct, this means things like otherwise
common situations such as a new block will result in disconnects, say when
the sender doesn't hear about a new block which makes the relay package
superfluous/irrelevant. Similar would be disconnection
when confirmed gets turned into unconfirmed, but those situations are
extremely uncommon. The other rules are entirely under the control
of the sender, which leads me to wonder if it's appropriate.

Cheers,
Greg

On Tue, May 17, 2022 at 12:09 PM Gloria Zhao via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> wrote:

> Hi everybody,
>
> I’m writing to propose a set of p2p protocol changes to enable package
> relay, soliciting feedback on the design and approach. Here is a link
> to the most up-to-date proposal:
>
> https://github.com/bitcoin/bips/pull/1324
>
> If you have concept or approach feedback, *please respond on the
> mailing list* to allow everybody to view and participate in the
> discussion. If you find a typo or inaccurate wording, please feel free
> to leave suggestions on the PR.
>
> I’m also working on an implementation for Bitcoin Core.
>
>
> The rest of this post will include the same contents as the proposal,
> with a bit of reordering and additional context. If you are not 100%
> up-to-date on package relay and find the proposal hard to follow, I
> hope you find this format more informative and persuasive.
>
>
> ==Background and Motivation==
>
> Users may create and broadcast transactions that depend upon, i.e.
> spend outputs of, unconfirmed transactions. A “package” is the
> widely-used term for a group of transactions representable by a
> connected Directed Acyclic Graph (where a directed edge exists between
> a transaction that spends the output of another transaction).
>
> Incentive-compatible mempool and miner policies help create a fair,
> fee-based market for block space. While miners maximize transaction
> fees in order to earn higher block rewards, non-mining users
> participating in transaction relay reap many benefits from employing
> policies that result in a mempool with the same contents, including
> faster compact block relay and more accurate fee estimation.
> Additionally, users may take advantage of mempool and miner policy to
> bump the priority of their transactions by attaching high-fee
> descendants (Child Pays for Parent or CPFP).  Only considering
> transactions one at a time for submission to the mempool creates a
> limitation in the node's ability to determine which transactions have
> the highest feerates, since it cannot take into account descendants
> until all the transactions are in the mempool. Similarly, it cannot
> use a transaction's descendants when considering which of two
> conflicting transactions to keep (Replace by Fee or RBF).
>
> When a user's transaction does not meet a mempool's minimum feerate
> and they cannot create a replacement transaction directly, their
> transaction will simply be rejected by this mempool. They also cannot
> attach a descendant to pay for replacing a conflicting transaction.
> This limitation harms users' ability to fee-bump their transactions.
> Further, it presents a security issue in contracting protocols which
> rely on **presigned**, time-sensitive transactions to prevent cheating
> (HTLC-Timeout in LN Penalty [1] [2] [3], Unvault Cancel in Revault
> [4], Refund Transaction in Discreet Log Contracts [5], Updates in
> eltoo [6]). In other words, a key security assumption of many
> contracting protocols is that all parties can propagate and confirm
> transactions in a timely manner.
>
> In the past few years, increasing attention [0][1][2][3][6] has been
> brought to **pinning attacks**, a type of censorship in which the
> attacker uses mempool policy restrictions to prevent a transaction
> from being relayed or getting mined.  TLDR: revocation transactions
> must meet a certain confirmation target to be effective, but their
> feerates are negotiated well ahead of broadcast time. If the
> forecasted feerate was too low and no fee-bumping options are
> available, attackers can steal money from their counterparties. I walk
> through a concrete example for stealing Lightning HTLC outputs at
> ~23:58 in this talk [7][8].  Note that most attacks are only possible
> when the market for blockspace at broadcast time  demands much higher
> feerates than originally anticipated at signing time. Always
> overestimating fees may sidestep this issue temporarily (while mempool
> traffic is low and predictable), but this solution is not foolproof
> and wastes users' money. The feerate market can change due to sudden
> spikes in traffic (e.g. huge 12sat/vB dump a few days ago [9]) or
> sustained, high volume of Bitcoin payments (e.g.  April 2021 and
> December 2017).
>
> The best solution is to enable nodes to consider packages of
> transactions as a unit, e.g. one or more low-fee parent transactions
> with a high-fee child, instead of separately. A package-aware mempool
> policy can help determine if it would actually be economically
> rational to accept a transaction to the mempool if it doesn't meet fee
> requirements individually. Network-wide adoption of these policies
> would create a more purely-feerate-based market for block space and
> allow contracting protocols to adjust fees (and therefore mining
> priority) at broadcast time.  Some support for packages has existed in
> Bitcoin Core for years. Since v0.13, Bitcoin Core has used ancestor
> packages instead of individual transactions to evaluate the incentive
> compatibility of transactions in the mempool [10] and select them for
> inclusion in blocks [11].
>
> Package Relay, the concept of {announcing, requesting, downloading}
> packages between nodes on the p2p network, has also been discussed for
> many years. The earliest public mention I can find is from 2015 [12].
> The two most common use cases for package relay are fee-bumping
> otherwise-too-low-fee transactions and reducing the amount of orphans.
> It seems uncontroversial to say that everybody desires package relay
> conceptually, with varying degrees of urgency. Lots of work has been
> done by others over the past few years, from which I've taken
> inspiration from [13][14][15][16].
>
> My approach has been to split the project into two components: (1) Package
> Mempool Accept, which includes validation logic and mempool policy.
> (3) Package Relay, which includes the p2p protocol changes.
>
> Progress so far:
> After discussions with various developers of contracting protocols
> (with heavier emphasis towards LN), it was determined that a
> package containing a child with all of its unconfirmed parents
> (child-with-unconfirmed-parents or 1-child-multi-parent package) would
> be sufficient for their use case, i.e. fee-bumping presigned
> transactions. A child-with-unconfirmed-parents package has several
> properties that make many things easier to reason about.
>
> A few months ago, I proposed a set of policies for safe package
> validation and fee assessment for packages of this restricted
> topology [17]. A series of PRs implementing this proposal have
> been merged into Bitcoin Core [18].
>
> Theoretically, developing a safe and incentive-compatible package
> mempool acceptance policy is sufficient to solve this issue. Nodes
> could opportunistically accept packages (e.g. by trying combinations
> of transactions rejected from their mempools), but this practice would
> likely be inefficient at best and open new Denial of Service attacks
> at worst. Additional p2p messages may enable nodes to request and
> share package validation-related information with one another in a
> more communication-efficient way.
>
> Given that only package RBF remains for package mempool accept, and we
> can make progress on p2p and mempool in parallel, I think it’s
> appropriate to put forward a package relay proposal.
>
> ==Proposal==
>
> This proposal contains 2 components: a “generic” package relay
> protocol and an extension of it, child-with-unconfirmed-parents
> packages, as version 1 package relay. Another version of packages,
> “tx-with-unconfirmed-ancestors” can be created to extend package relay
> for eliminating orphans.
>
> ===Generic Package Relay===
>
> Two main ideas are introduced:
>
> Download and validate packages of transactions together.
>
> Provide information to help peers decide whether to request and/or how
> to validate transactions which are part of a package.
>
> ====Intended Protocol Flow====
>
> Due to the asynchronous nature of a distributed transaction relay
> network, nodes may not receive all of the information needed to
> validate a transaction at once. For example, after a node completes
> Initial Block Download (IBD) and first starts participating in
> transaction relay with an empty mempool, it is common to receive
> orphans. In such scenarios where a node is aware that it is missing
> information, a ''receiver-initiated'' dialogue is appropriate:
>
> 1. Receiver requests package information.
>
> 2. The sender provides package information, including the wtxids of
>    the transactions in the package and anything else that might be
> relevant (e.g. total fees and size).
>
> 3. The reciever uses the package information to decide how to request
>    and validate the transactions.
>
> Sometimes, no matter what order transactions are received by a node,
> validating them individually is insufficient. When the sender is aware
> of additional information that the receiver needs to accept a package,
> a proactive ''sender-initiated'' dialogue should be enabled:
>
> 1. Sender announces they have package information pertaining to a
>    transaction that might otherwise be undesired on its own.
>
> 2. The receiver requests package information.
>
> 3. The sender provides package information, including the wtxids of
>    the transactions in the package and anything else that might be
> relevant (e.g. total fees and size).
>
> 4. The reciever uses the package information to decide how to request
>    and validate the transactions.
>
> Package relay is negotiated between two peers during the version
> handshake. Package relay requires both peers to support wtxid-based
> relay because package transactions are referenced by their wtxid.
>
> ====New Messages====
>
> Three new protocol messages are added for use in any version of
> package relay. Additionally, each version of package relay must define
> its own inv type and "pckginfo" message version, referred to in this
> document as "MSG_PCKG" and "pckginfo" respectively. See
> BIP-v1-packages for a concrete example.
>
> =====sendpackages=====
>
> {|
> |  Field Name  ||  Type  ||  Size  ||  Purpose
> |-
> |version || uint32_t || 4 || Denotes a package version supported by the
> node.
> |-
> |max_count || uint32_t || 4 ||Specifies the maximum number of transactions
> per package this node is
> willing to accept.
> |-
> |max_weight || uint32_t || 4 ||Specifies the maximum total weight per
> package this node is willing
> to accept.
> |-
> |}
>
> 1. The "sendpackages" message has the structure defined above, with
>    pchCommand == "sendpackages".
>
> 2. During version handshake, nodes should send a "sendpackages"
>    message indicate they support package relay and may request
> packages.
>
> 3. The message should contain a version supported by the node. Nodes
>    should send a "sendpackages" message for each version they support.
>
> 4. The "sendpackages" message MUST be sent before sending a "verack"
>    message. If a "sendpackages" message is received afer "verack", the
> sender should be disconnected.
>
> 5. If 'fRelay==false' in a peer's version message, the node must not
>    send "sendpackages" to them. If a "sendpackages" message is
> received by a peer after sending `fRelay==false` in their version
> message, the sender should be disconnected.
>
> 6.. Upon receipt of a "sendpackages" message with a version that is
> not supported, a node must treat the peer as if it never received the
> message.
>
> 7. If both peers send "wtxidrelay" and "sendpackages" with the same
>    version, the peers should announce, request, and send package
> information to each other.
>
> =====getpckgtxns=====
>
> {|
> |  Field Name  ||  Type  ||  Size  ||   Purpose
> |-
> |txns_length||CompactSize||1 or 3 bytes|| The number of transactions
> requested.
> |-
> |txns||List of wtxids||txns_length * 32|| The wtxids of each transaction
> in the package.
> |}
>
> 1. The "getpckgtxns" message has the structure defined above, with
>    pchCommand == "getpckgtxns".
>
> 2. A "getpckgtxns" message should be used to request all or some of
>    the transactions previously announced in a "pckginfo" message,
> specified by witness transactiosome id.
>
> 3. Upon receipt of a "getpckgtxns" message, a node must respond with
>    either a "pckgtxns" containing the requested transactions or a
> "notfound" message indicating one or more of the transactions is
> unavailable. This allows the receiver to avoid downloading and storing
> transactions that cannot be validated immediately.
>
> 4. A "getpckgtxns" message should only be sent if both peers agreed to
>    send packages in the version handshake. If a "getpckgtxns" message
> is received from a peer with which package relay was not negotiated,
> the sender should be disconnected.
>
> =====pckgtxns=====
>
> {|
> |  Field Name  ||  Type  ||  Size  ||   Purpose
> |-
> |txns_length||CompactSize||1 or 3 bytes|| The number of transactions
> provided.
> |-
> |txns||List of transactions||variable|| The transactions in the package.
> |}
>
> 1. The "pckgtxns" message has the structure defined above, with
>    pchCommand == "pckgtxns".
>
> 2. A "pckgtxns" message should contain the transaction data requested
>    using "getpckgtxns".
>
> 3. A "pckgtxns" message should only be sent to a peer that requested
>    the package using "getpckgtxns". If a node receives an unsolicited
> package, the sender should be disconnected.
>
> 4. A "pckgtxns" message should only be sent if both peers agreed to
>    send packages in the version handshake. If a "pckgtxns" message is
> received from a peer with which package relay was not negotiated, the
> sender should be disconnected.
>
> ===Version 1 Packages: child-with-unconfirmed-parents===
>
> This extends package relay for packages consisting of one transaction
> and all of its unconfirmed parents,by defining version 1 packages, a
> pckginfo1 message, and a MSG_PCKG1 inv type. It enables the use case
> in which a child pays for its otherwise-too-low-fee parents and their
> mempool conflict(s).
>
> ====Intended Protocol Flow====
>
> When relaying a package of low-fee parent(s) and high-fee child, the
> sender and receiver do the following:
>
> 1. Sender announces they have a child-with-unconfirmed-parents package
>    for a child that pays for otherwise-too-low-fee parent(s) using
> "inv(MSG_PCKG1)".
>
> 2. The receiver requests package information using
>    "getdata(MSG_PCKG1)".
>
> 3. The sender provides package information using "pckginfo1",
>    including the blockhash of the sender's best block, the wtxids of
> the transactions in the package, their total fees and total weight.
>
> 4. The reciever uses the package information to decide how to request
>    the transactions. For example, if the receiver already has some of
> the transactions in their mempool, they only request the missing ones.
> They could also decide not to request the package at all based on the
> fee information provided.
>
> 5. Upon receiving a "pckgtxns", the receiver submits the transactions
>    together as a package.
>
> ====New Messages====
>
> A new inv type, "MSG_PCKG1", and new protocol message, "PCKGINFO1",
> are added.
>
> =====pckginfo1=====
>
> {|
> |  Field Name  ||  Type  ||  Size  ||   Purpose
> |-
> |blockhash || uint256 || 32 || The chain tip at which this package is
> defined.
> |-
> |pckg_fee||CAmount||4|| The sum total fees paid by all transactions in the
> package.
> |-
> |pckg_weight||int64_t||8|| The sum total weight of all transactions in the
> package.
> |-
> |txns_length||CompactSize||1 or 3 bytes|| The number of transactions
> provided.
> |-
> |txns||List of wtxids||txns_length * 32|| The wtxids of each transaction
> in the package.
> |}
>
>
> 1. The "pckginfo1" message has the structure defined above, with
>    pchCommand == "pckginfo1".
>
> 2. A "pckginfo1" message contains information about a version 1
>    package (defined below), referenced by the wtxid of the transaction
> it pertains to and the current blockhash.
>
> 3. Upon receipt of a "pckginfo1" message, the node should decide if it
>    wants to validate the package, request transaction data if
> necessary, etc.
>
> 4. Upon receipt of a malformed "pckginfo1" message or package that
>    does not abide by the max_count, max_weight, or other rules
> specified by the version agreed upon in the initial negotiation, the
> sender should be disconnected.  If a node receives a "pckginfo1"
> message for which the "pckg_fee" or "pckg_weight" do not reflect the
> true total fees and weight, respectively, or the transactions in the
> package, the message is malformed.
>
> 5. A node MUST NOT send a "pckginfo1" message that has not been
>    requested by the recipient. Upon receipt of an unsolicited
> "pckginfo1", a node should disconnect the sender.
>
> 6. A "pckginfo1" message should only be sent if both peers agreed to
>    send version 1 packages in the version handshake. If a "pckginfo1"
> message is received from a peer with which package relay was not
> negotiated, the sender should be disconnected.
>
> =====MSG_PCKG1=====
>
> 1. A new inv type (MSG_PCKG1 == 0x6) is added, for use in inv messages
>    and getdata requests pertaining to version 1 packages.
>
> 2. As an inv type, it indicates that both transaction data and version
>    1 package information are available for the transaction. The
> transaction is referenced by its wtxid. As a getdata request type, it
> indicates that the sender wants package information for the
> transaction.
>
> 3. Upon receipt of a "getdata" request for "MSG_PCKG1", the node
>    should respond with the version 1 package corresponding to the
> requested transaction and its current chain tip, or with NOTFOUND.
> The node should not assume that the sender is requesting the
> transaction data as well.
>
> ====Child With Parent Packages Rules====
>
> A child-with-unconfirmed-parents package sent between nodes must abide
> by the rules below, otherwise the package is malformed and the sender
> should be disconnected.
>
> A version 1 or ''child-with-unconfirmed-parents'' package can be
> defined for any transaction that spends unconfirmed inputs. The child
> can be thought of as the "representative" of the package. This package
> can be uniquely identified by the transaction's wtxid and the current
> chain tip block hash.
>
> A ''child-with-unconfirmed-parents'' package MUST be:
>
> 1. ''Sorted topologically.'' For every transaction t in the package,
>    if any of t's parents are present in the package, the parent must
> appear somewhere in the list before t. In other words, the
> transactions must be sorted in ascending order of the number of
> ancestors present in the package.
>
> 2. ''Only 1 child with unconfirmed parents.'' The package must consist
>    of one transaction and its unconfirmed parents. There must not be
> any other transactions in the package. Other dependency relationships
> may exist within the package (e.g. one parent may spend the output of
> another parent) provided that topological order is respected.
>
> 3. ''All unconfirmed parents.'' All of the child's unconfirmed parents
>    must be present.
>
> 4. ''No conflicts.'' None of the transactions in the package may
>    conflict with each other (i.e.  spend the same prevout).
>
> 5. ''Total fees and weight.'' The 'total_fee' and 'total_weight'
>    fields must accurately represent the sum total of all transactions'
> fees and weights as defined in BIP141, respectively.
>
> Not all of the child's parents must be present; the child transaction
> may also spend confirmed inputs. However, if the child has confirmed
> parents, they must not be in the package.
>
> While a child-with-unconfirmed-parents package is perhaps most
> relevant when the child has a higher feerate than its parents, this
> property is not required to construct a valid package.
>
> ====Clarifications====
>
> ''Q: Under what circumstances should a sender announce a
> child-with-unconfirmed-parents package?''
>
> A child-with-unconfirmed-parents package for a transaction should be
> announced when it meets the peer's fee filter but one or more of its
> parents don't; a "inv(MSG_PCKG1)" instead of "inv(WTX)" should be sent
> for the child. Each of the parents which meet the peer's fee filter
> should still be announced normally.
>
> ''Q: What if a new block arrives in between messages?''
>
> A child-with-unconfirmed-parents package is defined for a transaction
> based on the current chain state. As such, a new block extending the
> tip may decrease the number of transactions in the package (i.e. if
> any of the transaction's parents were included in the block). In a
> reorg, the number of transactions in the package may decrease or
> increase (i.e. if any of the transaction's parents were included in a
> block in the previous chain but not the new one).
>
> If the new block arrives before the "getdata" or "pckginfo1", nothing
> needs to change.
>
> If the new block arrives before "getpckgtxns" or before "pckgtxns",
> the receiver may need to re-request package information if the block
> contained a transaction in the package. If the block doesn't contain
> any transactions in the package, whether it extends the previous tip
> or causes a reorg, nothing needs to change.
>
> ''Q: Can "getpckgtxns" and "pckgtxns" messages contain only one
> transaction?''
>
> Yes.
>
> ===Further Protocol Extensions===
>
> When introducing a new type of package, assign it a version number "n"
> and use an additional "sendpackages" message during version handshake
> to negotiate support for it. An additional package information message
> "pckginfon" and inv type "MSG_PCKGn" should be defined for the type of
> package.  However, "getpckgtxns" and "pckgtxns" do not need to be
> changed.
>
> Example proposal for tx-with-unconfirmed-ancestors package relay: [19]
>
> ===Compatibility===
>
> Older clients remain fully compatible and interoperable after this
> change. Clients implementing this protocol will only attempt to send
> and request packages if agreed upon during the version handshake.
>
> ===Package Erlay===
>
> Clients using BIP330 reconciliation-based transaction relay (Erlay)
> are able to use package relay without interference. In fact, a package
> of transactions may be announced using both Erlay and package relay.
> After reconciliation, if the initiator would have announced a
> transaction by wtxid but also has package information for it, they may
> send "inv(MSG_PCKG)" instead of "inv(WTX)".
>
> ===Rationale===
>
> ====P2P Message Design====
>
> These p2p messages are added for communication efficiency and, as
> such, one should measure alternative solutions based on the resources
> used to communicate (not necessarily trustworthy) information: We
> would like to minimize network bandwidth, avoid downloading a
> transaction more than once, avoid downloading transactions that are
> eventually rejected, and minimize storage allocated for
> not-yet-validated transactions.
>
> Consider these (plausible) scenarios in transaction relay:
>
> Alice (the "sender") is relaying transactions to Bob (the "receiver").
> Alice's mempool has a minimum feerate of 1sat/vB and Bob's has a
> minimum feerate of 3sat/vB. For simplicity, all transactions are
> 1600Wu in virtual size and 500 bytes in serialized size. Apart from
> the spending relationships specified, all other inputs are from
> confirmed UTXOs.
>
> 1. Package {A, B} where A pays 0 satoshis and B pays 8000 satoshis in
>    fees.
>
> 2. Package {C, D} where C pays 0 satoshis and D pays 1200 satoshis in
>    fees.
>
> 3. Package {E, F, G, H, J} that pays 4000, 8000, 0, 2000, and 4000
>    satoshis in fees, respectively.
>
> ====Alternative Designs Considered====
>
> ''Package Information Only:'' Just having "pckginfo" gives enough
> information for the receiver to accept the package. Omit the
> "getpckgtxns" and "pckgtxns" messages. While this option is a good
> fallback if batched transaction download fails for some reason, it
> shouldn't be used as the default because it 'always' requires storage
> of unvalidated transactions.
>
> ''No Package Information Round:'' Instead of having a package
> information round, just use the child's wtxid to refer to the package
> and always send the entire package together. This would cause nodes to
> redownload duplicate transactions.
>
> I have also created a slidedeck exploring various alternative designs
> and some examples in which they fall flat [20]. Please feel free to
> suggest other alternatives.
>
> ====Versioning System====
>
> This protocol should be extensible to support multiple types of
> packages based on future desired use cases. Two "flavors" of
> versioning were considered:
>
> 1. When package mempool acceptance is upgraded to support more types
>    of packages, increment the version number (similar to Erlay).
> During version handshake, peers negotiate which version of package
> relay they will use by each sending one "sendpackages" message.
>
> 2. When introducing another type of package, assign a version number
>    to it and announce it as an additional supported version (similar
> to Compact Block Relay). During version handshake, peers send one
> "sendpackages" message for each version supported.
>
> The second option was favored because it allows different parameters
> for different versions.  For example, it should be possible to support
> both "arbitrary topology but maximum 3-transaction" package as well as
> "child-with-unconfirmed-parents with default mempool ancestor limits"
> packages simultaneously.
>
> ==Acknowledgements==
>
> I hope to have made it abundantly clear that this proposal isn’t
> inventing the concept of package relay, and in fact builds upon years
> of work by many others, including Suhas Daftuar and Antoine Riard.
>
> Thank you to John Newbery and Martin Zumsande for input on the design.
>
> Thank you to Matt Corallo, Christian Decker, David Harding, Antoine
> Poinsot, Antoine Riard, Gregory Sanders, Chris Stewart, Bastien
> Teinturier, and others for input on the desired interface for
> contracting protocols.
>
> Looking forward to hearing your thoughts!
>
> Best,
> Gloria
>
> [0]:
> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-January/019817.html
> [1]:
> https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-April/002639.html
> [2]:
> https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-June/002758.html
> [3]:
> https://github.com/t-bast/lightning-docs/blob/master/pinning-attacks.md
> [4]:
> https://github.com/revault/practical-revault/blob/master/transactions.md#cancel_tx
> [5]:
> https://github.com/discreetlogcontracts/dlcspecs/blob/master/Transactions.md#refund-transaction
> [6]: https://gist.github.com/instagibbs/60264606e181451e977e439a49f69fe1
> [7]:
> https://btctranscripts.com/adopting-bitcoin/2021/2021-11-16-gloria-zhao-transaction-relay-policy/#lightning-attacks
> [8]: https://youtu.be/fbWSQvJjKFs?t=1438
> [9]:
> https://www.reddit.com/r/Bitcoin/comments/unew4e/looks_like_70_mvb_of_transactions_just_got_dumped/
> [10]: https://github.com/bitcoin/bitcoin/pull/7594
> [11]: https://github.com/bitcoin/bitcoin/pull/7600
> [12]: https://github.com/bitcoin/bitcoin/pull/6455#issuecomment-122716820
> [13]: https://gist.github.com/sdaftuar/8756699bfcad4d3806ba9f3396d4e66a
> [14]: https://github.com/bitcoin/bitcoin/issues/14895
> [15]: https://github.com/bitcoin/bitcoin/pull/16401
> [16]: https://github.com/bitcoin/bitcoin/pull/19621
> [17]:
> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-September/019464.html
> [18]: https://github.com/users/glozow/projects/5/views/4?layout=board
> [19]: https://gist.github.com/glozow/9b321cd3ef6505135c763112033ff2a7
> [20]:
> https://docs.google.com/presentation/d/1B__KlZO1VzxJGx-0DYChlWawaEmGJ9EGApEzrHqZpQc/edit?usp=sharing
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 34756 bytes --]

 permalink raw reply	[|nested]
2022-05-17 17:56 ` Greg Sanders

  
From: Gloria Zhao @ 2022-05-17 20:45 UTC (permalink / raw)
  To: Greg Sanders;  Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 32318 bytes --]

Hi Greg,

Thanks for reading!

>> A child-with-unconfirmed-parents package sent between nodes must abide by
the rules below, otherwise the package is malformed and the sender should
be disconnected.

>> However, if the child has confirmed parents, they must not be in the
package.

> If my naive understanding is correct, this means things like otherwise
common situations such as a new block will result in disconnects, say when
> the sender doesn't hear about a new block which makes the relay package
superfluous/irrelevant. Similar would be disconnection
> when confirmed gets turned into unconfirmed, but those situations are
extremely uncommon. The other rules are entirely under the control
> of the sender, which leads me to wonder if it's appropriate.

This is why the "pckginfo1" message includes the blockhash at which the
package was defined.
Also please see Clarifications - "Q: What if a new block arrives in between
messages?'' section in the v1-packages portion. It covers both cases, i.e.
a transaction going from unconfirmed->confirmed and confirmed->unconfirmed
in a reorg.

In case anybody is wondering "why don't we just allow confirmed parents?":
Since we validate based on the UTXO set, when we see a recently-confirmed
transaction, it just looks like it spends nonexistent inputs. In these
cases, we don't really know if the input was recently spent in a block or
just never existed, unless we plan on looking up transactions in past
blocks. We do some guesswork when we deal with new blocks in normal
transaction relay (e.g. we requested the tx before a block arrived):
https://github.com/bitcoin/bitcoin/blob/d5d40d59f8d12cf53c5ad1ce9710f3f108cec386/src/validation.cpp#L780-L784
I believe it's cleaner to just explicitly say which blockhash you're on to
avoid confusion.

Thanks,
Gloria

On Tue, May 17, 2022 at 1:56 PM Greg Sanders <gsanders87@gmail•com> wrote:

> Hi Gloria,
>
> Thanks for working on this important proposal!
>
> Still a lot to digest, but I just had on area of comment/question:
>
> > A child-with-unconfirmed-parents package sent between nodes must abide by
> the rules below, otherwise the package is malformed and the sender should
> be disconnected.
>
> > However, if the child has confirmed parents, they must not be in the
> package.
>
> If my naive understanding is correct, this means things like otherwise
> common situations such as a new block will result in disconnects, say when
> the sender doesn't hear about a new block which makes the relay package
> superfluous/irrelevant. Similar would be disconnection
> when confirmed gets turned into unconfirmed, but those situations are
> extremely uncommon. The other rules are entirely under the control
> of the sender, which leads me to wonder if it's appropriate.
>
> Cheers,
> Greg
>
> On Tue, May 17, 2022 at 12:09 PM Gloria Zhao via bitcoin-dev <
> bitcoin-dev@lists•linuxfoundation.org> wrote:
>
>> Hi everybody,
>>
>> I’m writing to propose a set of p2p protocol changes to enable package
>> relay, soliciting feedback on the design and approach. Here is a link
>> to the most up-to-date proposal:
>>
>> https://github.com/bitcoin/bips/pull/1324
>>
>> If you have concept or approach feedback, *please respond on the
>> mailing list* to allow everybody to view and participate in the
>> discussion. If you find a typo or inaccurate wording, please feel free
>> to leave suggestions on the PR.
>>
>> I’m also working on an implementation for Bitcoin Core.
>>
>>
>> The rest of this post will include the same contents as the proposal,
>> with a bit of reordering and additional context. If you are not 100%
>> up-to-date on package relay and find the proposal hard to follow, I
>> hope you find this format more informative and persuasive.
>>
>>
>> ==Background and Motivation==
>>
>> Users may create and broadcast transactions that depend upon, i.e.
>> spend outputs of, unconfirmed transactions. A “package” is the
>> widely-used term for a group of transactions representable by a
>> connected Directed Acyclic Graph (where a directed edge exists between
>> a transaction that spends the output of another transaction).
>>
>> Incentive-compatible mempool and miner policies help create a fair,
>> fee-based market for block space. While miners maximize transaction
>> fees in order to earn higher block rewards, non-mining users
>> participating in transaction relay reap many benefits from employing
>> policies that result in a mempool with the same contents, including
>> faster compact block relay and more accurate fee estimation.
>> Additionally, users may take advantage of mempool and miner policy to
>> bump the priority of their transactions by attaching high-fee
>> descendants (Child Pays for Parent or CPFP).  Only considering
>> transactions one at a time for submission to the mempool creates a
>> limitation in the node's ability to determine which transactions have
>> the highest feerates, since it cannot take into account descendants
>> until all the transactions are in the mempool. Similarly, it cannot
>> use a transaction's descendants when considering which of two
>> conflicting transactions to keep (Replace by Fee or RBF).
>>
>> When a user's transaction does not meet a mempool's minimum feerate
>> and they cannot create a replacement transaction directly, their
>> transaction will simply be rejected by this mempool. They also cannot
>> attach a descendant to pay for replacing a conflicting transaction.
>> This limitation harms users' ability to fee-bump their transactions.
>> Further, it presents a security issue in contracting protocols which
>> rely on **presigned**, time-sensitive transactions to prevent cheating
>> (HTLC-Timeout in LN Penalty [1] [2] [3], Unvault Cancel in Revault
>> [4], Refund Transaction in Discreet Log Contracts [5], Updates in
>> eltoo [6]). In other words, a key security assumption of many
>> contracting protocols is that all parties can propagate and confirm
>> transactions in a timely manner.
>>
>> In the past few years, increasing attention [0][1][2][3][6] has been
>> brought to **pinning attacks**, a type of censorship in which the
>> attacker uses mempool policy restrictions to prevent a transaction
>> from being relayed or getting mined.  TLDR: revocation transactions
>> must meet a certain confirmation target to be effective, but their
>> feerates are negotiated well ahead of broadcast time. If the
>> forecasted feerate was too low and no fee-bumping options are
>> available, attackers can steal money from their counterparties. I walk
>> through a concrete example for stealing Lightning HTLC outputs at
>> ~23:58 in this talk [7][8].  Note that most attacks are only possible
>> when the market for blockspace at broadcast time  demands much higher
>> feerates than originally anticipated at signing time. Always
>> overestimating fees may sidestep this issue temporarily (while mempool
>> traffic is low and predictable), but this solution is not foolproof
>> and wastes users' money. The feerate market can change due to sudden
>> spikes in traffic (e.g. huge 12sat/vB dump a few days ago [9]) or
>> sustained, high volume of Bitcoin payments (e.g.  April 2021 and
>> December 2017).
>>
>> The best solution is to enable nodes to consider packages of
>> transactions as a unit, e.g. one or more low-fee parent transactions
>> with a high-fee child, instead of separately. A package-aware mempool
>> policy can help determine if it would actually be economically
>> rational to accept a transaction to the mempool if it doesn't meet fee
>> requirements individually. Network-wide adoption of these policies
>> would create a more purely-feerate-based market for block space and
>> allow contracting protocols to adjust fees (and therefore mining
>> priority) at broadcast time.  Some support for packages has existed in
>> Bitcoin Core for years. Since v0.13, Bitcoin Core has used ancestor
>> packages instead of individual transactions to evaluate the incentive
>> compatibility of transactions in the mempool [10] and select them for
>> inclusion in blocks [11].
>>
>> Package Relay, the concept of {announcing, requesting, downloading}
>> packages between nodes on the p2p network, has also been discussed for
>> many years. The earliest public mention I can find is from 2015 [12].
>> The two most common use cases for package relay are fee-bumping
>> otherwise-too-low-fee transactions and reducing the amount of orphans.
>> It seems uncontroversial to say that everybody desires package relay
>> conceptually, with varying degrees of urgency. Lots of work has been
>> done by others over the past few years, from which I've taken
>> inspiration from [13][14][15][16].
>>
>> My approach has been to split the project into two components: (1) Package
>> Mempool Accept, which includes validation logic and mempool policy.
>> (3) Package Relay, which includes the p2p protocol changes.
>>
>> Progress so far:
>> After discussions with various developers of contracting protocols
>> (with heavier emphasis towards LN), it was determined that a
>> package containing a child with all of its unconfirmed parents
>> (child-with-unconfirmed-parents or 1-child-multi-parent package) would
>> be sufficient for their use case, i.e. fee-bumping presigned
>> transactions. A child-with-unconfirmed-parents package has several
>> properties that make many things easier to reason about.
>>
>> A few months ago, I proposed a set of policies for safe package
>> validation and fee assessment for packages of this restricted
>> topology [17]. A series of PRs implementing this proposal have
>> been merged into Bitcoin Core [18].
>>
>> Theoretically, developing a safe and incentive-compatible package
>> mempool acceptance policy is sufficient to solve this issue. Nodes
>> could opportunistically accept packages (e.g. by trying combinations
>> of transactions rejected from their mempools), but this practice would
>> likely be inefficient at best and open new Denial of Service attacks
>> at worst. Additional p2p messages may enable nodes to request and
>> share package validation-related information with one another in a
>> more communication-efficient way.
>>
>> Given that only package RBF remains for package mempool accept, and we
>> can make progress on p2p and mempool in parallel, I think it’s
>> appropriate to put forward a package relay proposal.
>>
>> ==Proposal==
>>
>> This proposal contains 2 components: a “generic” package relay
>> protocol and an extension of it, child-with-unconfirmed-parents
>> packages, as version 1 package relay. Another version of packages,
>> “tx-with-unconfirmed-ancestors” can be created to extend package relay
>> for eliminating orphans.
>>
>> ===Generic Package Relay===
>>
>> Two main ideas are introduced:
>>
>> Download and validate packages of transactions together.
>>
>> Provide information to help peers decide whether to request and/or how
>> to validate transactions which are part of a package.
>>
>> ====Intended Protocol Flow====
>>
>> Due to the asynchronous nature of a distributed transaction relay
>> network, nodes may not receive all of the information needed to
>> validate a transaction at once. For example, after a node completes
>> Initial Block Download (IBD) and first starts participating in
>> transaction relay with an empty mempool, it is common to receive
>> orphans. In such scenarios where a node is aware that it is missing
>> information, a ''receiver-initiated'' dialogue is appropriate:
>>
>> 1. Receiver requests package information.
>>
>> 2. The sender provides package information, including the wtxids of
>>    the transactions in the package and anything else that might be
>> relevant (e.g. total fees and size).
>>
>> 3. The reciever uses the package information to decide how to request
>>    and validate the transactions.
>>
>> Sometimes, no matter what order transactions are received by a node,
>> validating them individually is insufficient. When the sender is aware
>> of additional information that the receiver needs to accept a package,
>> a proactive ''sender-initiated'' dialogue should be enabled:
>>
>> 1. Sender announces they have package information pertaining to a
>>    transaction that might otherwise be undesired on its own.
>>
>> 2. The receiver requests package information.
>>
>> 3. The sender provides package information, including the wtxids of
>>    the transactions in the package and anything else that might be
>> relevant (e.g. total fees and size).
>>
>> 4. The reciever uses the package information to decide how to request
>>    and validate the transactions.
>>
>> Package relay is negotiated between two peers during the version
>> handshake. Package relay requires both peers to support wtxid-based
>> relay because package transactions are referenced by their wtxid.
>>
>> ====New Messages====
>>
>> Three new protocol messages are added for use in any version of
>> package relay. Additionally, each version of package relay must define
>> its own inv type and "pckginfo" message version, referred to in this
>> document as "MSG_PCKG" and "pckginfo" respectively. See
>> BIP-v1-packages for a concrete example.
>>
>> =====sendpackages=====
>>
>> {|
>> |  Field Name  ||  Type  ||  Size  ||  Purpose
>> |-
>> |version || uint32_t || 4 || Denotes a package version supported by the
>> node.
>> |-
>> |max_count || uint32_t || 4 ||Specifies the maximum number of
>> transactions per package this node is
>> willing to accept.
>> |-
>> |max_weight || uint32_t || 4 ||Specifies the maximum total weight per
>> package this node is willing
>> to accept.
>> |-
>> |}
>>
>> 1. The "sendpackages" message has the structure defined above, with
>>    pchCommand == "sendpackages".
>>
>> 2. During version handshake, nodes should send a "sendpackages"
>>    message indicate they support package relay and may request
>> packages.
>>
>> 3. The message should contain a version supported by the node. Nodes
>>    should send a "sendpackages" message for each version they support.
>>
>> 4. The "sendpackages" message MUST be sent before sending a "verack"
>>    message. If a "sendpackages" message is received afer "verack", the
>> sender should be disconnected.
>>
>> 5. If 'fRelay==false' in a peer's version message, the node must not
>>    send "sendpackages" to them. If a "sendpackages" message is
>> received by a peer after sending `fRelay==false` in their version
>> message, the sender should be disconnected.
>>
>> 6.. Upon receipt of a "sendpackages" message with a version that is
>> not supported, a node must treat the peer as if it never received the
>> message.
>>
>> 7. If both peers send "wtxidrelay" and "sendpackages" with the same
>>    version, the peers should announce, request, and send package
>> information to each other.
>>
>> =====getpckgtxns=====
>>
>> {|
>> |  Field Name  ||  Type  ||  Size  ||   Purpose
>> |-
>> |txns_length||CompactSize||1 or 3 bytes|| The number of transactions
>> requested.
>> |-
>> |txns||List of wtxids||txns_length * 32|| The wtxids of each transaction
>> in the package.
>> |}
>>
>> 1. The "getpckgtxns" message has the structure defined above, with
>>    pchCommand == "getpckgtxns".
>>
>> 2. A "getpckgtxns" message should be used to request all or some of
>>    the transactions previously announced in a "pckginfo" message,
>> specified by witness transactiosome id.
>>
>> 3. Upon receipt of a "getpckgtxns" message, a node must respond with
>>    either a "pckgtxns" containing the requested transactions or a
>> "notfound" message indicating one or more of the transactions is
>> unavailable. This allows the receiver to avoid downloading and storing
>> transactions that cannot be validated immediately.
>>
>> 4. A "getpckgtxns" message should only be sent if both peers agreed to
>>    send packages in the version handshake. If a "getpckgtxns" message
>> is received from a peer with which package relay was not negotiated,
>> the sender should be disconnected.
>>
>> =====pckgtxns=====
>>
>> {|
>> |  Field Name  ||  Type  ||  Size  ||   Purpose
>> |-
>> |txns_length||CompactSize||1 or 3 bytes|| The number of transactions
>> provided.
>> |-
>> |txns||List of transactions||variable|| The transactions in the package.
>> |}
>>
>> 1. The "pckgtxns" message has the structure defined above, with
>>    pchCommand == "pckgtxns".
>>
>> 2. A "pckgtxns" message should contain the transaction data requested
>>    using "getpckgtxns".
>>
>> 3. A "pckgtxns" message should only be sent to a peer that requested
>>    the package using "getpckgtxns". If a node receives an unsolicited
>> package, the sender should be disconnected.
>>
>> 4. A "pckgtxns" message should only be sent if both peers agreed to
>>    send packages in the version handshake. If a "pckgtxns" message is
>> received from a peer with which package relay was not negotiated, the
>> sender should be disconnected.
>>
>> ===Version 1 Packages: child-with-unconfirmed-parents===
>>
>> This extends package relay for packages consisting of one transaction
>> and all of its unconfirmed parents,by defining version 1 packages, a
>> pckginfo1 message, and a MSG_PCKG1 inv type. It enables the use case
>> in which a child pays for its otherwise-too-low-fee parents and their
>> mempool conflict(s).
>>
>> ====Intended Protocol Flow====
>>
>> When relaying a package of low-fee parent(s) and high-fee child, the
>> sender and receiver do the following:
>>
>> 1. Sender announces they have a child-with-unconfirmed-parents package
>>    for a child that pays for otherwise-too-low-fee parent(s) using
>> "inv(MSG_PCKG1)".
>>
>> 2. The receiver requests package information using
>>    "getdata(MSG_PCKG1)".
>>
>> 3. The sender provides package information using "pckginfo1",
>>    including the blockhash of the sender's best block, the wtxids of
>> the transactions in the package, their total fees and total weight.
>>
>> 4. The reciever uses the package information to decide how to request
>>    the transactions. For example, if the receiver already has some of
>> the transactions in their mempool, they only request the missing ones.
>> They could also decide not to request the package at all based on the
>> fee information provided.
>>
>> 5. Upon receiving a "pckgtxns", the receiver submits the transactions
>>    together as a package.
>>
>> ====New Messages====
>>
>> A new inv type, "MSG_PCKG1", and new protocol message, "PCKGINFO1",
>> are added.
>>
>> =====pckginfo1=====
>>
>> {|
>> |  Field Name  ||  Type  ||  Size  ||   Purpose
>> |-
>> |blockhash || uint256 || 32 || The chain tip at which this package is
>> defined.
>> |-
>> |pckg_fee||CAmount||4|| The sum total fees paid by all transactions in
>> the package.
>> |-
>> |pckg_weight||int64_t||8|| The sum total weight of all transactions in
>> the package.
>> |-
>> |txns_length||CompactSize||1 or 3 bytes|| The number of transactions
>> provided.
>> |-
>> |txns||List of wtxids||txns_length * 32|| The wtxids of each transaction
>> in the package.
>> |}
>>
>>
>> 1. The "pckginfo1" message has the structure defined above, with
>>    pchCommand == "pckginfo1".
>>
>> 2. A "pckginfo1" message contains information about a version 1
>>    package (defined below), referenced by the wtxid of the transaction
>> it pertains to and the current blockhash.
>>
>> 3. Upon receipt of a "pckginfo1" message, the node should decide if it
>>    wants to validate the package, request transaction data if
>> necessary, etc.
>>
>> 4. Upon receipt of a malformed "pckginfo1" message or package that
>>    does not abide by the max_count, max_weight, or other rules
>> specified by the version agreed upon in the initial negotiation, the
>> sender should be disconnected.  If a node receives a "pckginfo1"
>> message for which the "pckg_fee" or "pckg_weight" do not reflect the
>> true total fees and weight, respectively, or the transactions in the
>> package, the message is malformed.
>>
>> 5. A node MUST NOT send a "pckginfo1" message that has not been
>>    requested by the recipient. Upon receipt of an unsolicited
>> "pckginfo1", a node should disconnect the sender.
>>
>> 6. A "pckginfo1" message should only be sent if both peers agreed to
>>    send version 1 packages in the version handshake. If a "pckginfo1"
>> message is received from a peer with which package relay was not
>> negotiated, the sender should be disconnected.
>>
>> =====MSG_PCKG1=====
>>
>> 1. A new inv type (MSG_PCKG1 == 0x6) is added, for use in inv messages
>>    and getdata requests pertaining to version 1 packages.
>>
>> 2. As an inv type, it indicates that both transaction data and version
>>    1 package information are available for the transaction. The
>> transaction is referenced by its wtxid. As a getdata request type, it
>> indicates that the sender wants package information for the
>> transaction.
>>
>> 3. Upon receipt of a "getdata" request for "MSG_PCKG1", the node
>>    should respond with the version 1 package corresponding to the
>> requested transaction and its current chain tip, or with NOTFOUND.
>> The node should not assume that the sender is requesting the
>> transaction data as well.
>>
>> ====Child With Parent Packages Rules====
>>
>> A child-with-unconfirmed-parents package sent between nodes must abide
>> by the rules below, otherwise the package is malformed and the sender
>> should be disconnected.
>>
>> A version 1 or ''child-with-unconfirmed-parents'' package can be
>> defined for any transaction that spends unconfirmed inputs. The child
>> can be thought of as the "representative" of the package. This package
>> can be uniquely identified by the transaction's wtxid and the current
>> chain tip block hash.
>>
>> A ''child-with-unconfirmed-parents'' package MUST be:
>>
>> 1. ''Sorted topologically.'' For every transaction t in the package,
>>    if any of t's parents are present in the package, the parent must
>> appear somewhere in the list before t. In other words, the
>> transactions must be sorted in ascending order of the number of
>> ancestors present in the package.
>>
>> 2. ''Only 1 child with unconfirmed parents.'' The package must consist
>>    of one transaction and its unconfirmed parents. There must not be
>> any other transactions in the package. Other dependency relationships
>> may exist within the package (e.g. one parent may spend the output of
>> another parent) provided that topological order is respected.
>>
>> 3. ''All unconfirmed parents.'' All of the child's unconfirmed parents
>>    must be present.
>>
>> 4. ''No conflicts.'' None of the transactions in the package may
>>    conflict with each other (i.e.  spend the same prevout).
>>
>> 5. ''Total fees and weight.'' The 'total_fee' and 'total_weight'
>>    fields must accurately represent the sum total of all transactions'
>> fees and weights as defined in BIP141, respectively.
>>
>> Not all of the child's parents must be present; the child transaction
>> may also spend confirmed inputs. However, if the child has confirmed
>> parents, they must not be in the package.
>>
>> While a child-with-unconfirmed-parents package is perhaps most
>> relevant when the child has a higher feerate than its parents, this
>> property is not required to construct a valid package.
>>
>> ====Clarifications====
>>
>> ''Q: Under what circumstances should a sender announce a
>> child-with-unconfirmed-parents package?''
>>
>> A child-with-unconfirmed-parents package for a transaction should be
>> announced when it meets the peer's fee filter but one or more of its
>> parents don't; a "inv(MSG_PCKG1)" instead of "inv(WTX)" should be sent
>> for the child. Each of the parents which meet the peer's fee filter
>> should still be announced normally.
>>
>> ''Q: What if a new block arrives in between messages?''
>>
>> A child-with-unconfirmed-parents package is defined for a transaction
>> based on the current chain state. As such, a new block extending the
>> tip may decrease the number of transactions in the package (i.e. if
>> any of the transaction's parents were included in the block). In a
>> reorg, the number of transactions in the package may decrease or
>> increase (i.e. if any of the transaction's parents were included in a
>> block in the previous chain but not the new one).
>>
>> If the new block arrives before the "getdata" or "pckginfo1", nothing
>> needs to change.
>>
>> If the new block arrives before "getpckgtxns" or before "pckgtxns",
>> the receiver may need to re-request package information if the block
>> contained a transaction in the package. If the block doesn't contain
>> any transactions in the package, whether it extends the previous tip
>> or causes a reorg, nothing needs to change.
>>
>> ''Q: Can "getpckgtxns" and "pckgtxns" messages contain only one
>> transaction?''
>>
>> Yes.
>>
>> ===Further Protocol Extensions===
>>
>> When introducing a new type of package, assign it a version number "n"
>> and use an additional "sendpackages" message during version handshake
>> to negotiate support for it. An additional package information message
>> "pckginfon" and inv type "MSG_PCKGn" should be defined for the type of
>> package.  However, "getpckgtxns" and "pckgtxns" do not need to be
>> changed.
>>
>> Example proposal for tx-with-unconfirmed-ancestors package relay: [19]
>>
>> ===Compatibility===
>>
>> Older clients remain fully compatible and interoperable after this
>> change. Clients implementing this protocol will only attempt to send
>> and request packages if agreed upon during the version handshake.
>>
>> ===Package Erlay===
>>
>> Clients using BIP330 reconciliation-based transaction relay (Erlay)
>> are able to use package relay without interference. In fact, a package
>> of transactions may be announced using both Erlay and package relay.
>> After reconciliation, if the initiator would have announced a
>> transaction by wtxid but also has package information for it, they may
>> send "inv(MSG_PCKG)" instead of "inv(WTX)".
>>
>> ===Rationale===
>>
>> ====P2P Message Design====
>>
>> These p2p messages are added for communication efficiency and, as
>> such, one should measure alternative solutions based on the resources
>> used to communicate (not necessarily trustworthy) information: We
>> would like to minimize network bandwidth, avoid downloading a
>> transaction more than once, avoid downloading transactions that are
>> eventually rejected, and minimize storage allocated for
>> not-yet-validated transactions.
>>
>> Consider these (plausible) scenarios in transaction relay:
>>
>> Alice (the "sender") is relaying transactions to Bob (the "receiver").
>> Alice's mempool has a minimum feerate of 1sat/vB and Bob's has a
>> minimum feerate of 3sat/vB. For simplicity, all transactions are
>> 1600Wu in virtual size and 500 bytes in serialized size. Apart from
>> the spending relationships specified, all other inputs are from
>> confirmed UTXOs.
>>
>> 1. Package {A, B} where A pays 0 satoshis and B pays 8000 satoshis in
>>    fees.
>>
>> 2. Package {C, D} where C pays 0 satoshis and D pays 1200 satoshis in
>>    fees.
>>
>> 3. Package {E, F, G, H, J} that pays 4000, 8000, 0, 2000, and 4000
>>    satoshis in fees, respectively.
>>
>> ====Alternative Designs Considered====
>>
>> ''Package Information Only:'' Just having "pckginfo" gives enough
>> information for the receiver to accept the package. Omit the
>> "getpckgtxns" and "pckgtxns" messages. While this option is a good
>> fallback if batched transaction download fails for some reason, it
>> shouldn't be used as the default because it 'always' requires storage
>> of unvalidated transactions.
>>
>> ''No Package Information Round:'' Instead of having a package
>> information round, just use the child's wtxid to refer to the package
>> and always send the entire package together. This would cause nodes to
>> redownload duplicate transactions.
>>
>> I have also created a slidedeck exploring various alternative designs
>> and some examples in which they fall flat [20]. Please feel free to
>> suggest other alternatives.
>>
>> ====Versioning System====
>>
>> This protocol should be extensible to support multiple types of
>> packages based on future desired use cases. Two "flavors" of
>> versioning were considered:
>>
>> 1. When package mempool acceptance is upgraded to support more types
>>    of packages, increment the version number (similar to Erlay).
>> During version handshake, peers negotiate which version of package
>> relay they will use by each sending one "sendpackages" message.
>>
>> 2. When introducing another type of package, assign a version number
>>    to it and announce it as an additional supported version (similar
>> to Compact Block Relay). During version handshake, peers send one
>> "sendpackages" message for each version supported.
>>
>> The second option was favored because it allows different parameters
>> for different versions.  For example, it should be possible to support
>> both "arbitrary topology but maximum 3-transaction" package as well as
>> "child-with-unconfirmed-parents with default mempool ancestor limits"
>> packages simultaneously.
>>
>> ==Acknowledgements==
>>
>> I hope to have made it abundantly clear that this proposal isn’t
>> inventing the concept of package relay, and in fact builds upon years
>> of work by many others, including Suhas Daftuar and Antoine Riard.
>>
>> Thank you to John Newbery and Martin Zumsande for input on the design.
>>
>> Thank you to Matt Corallo, Christian Decker, David Harding, Antoine
>> Poinsot, Antoine Riard, Gregory Sanders, Chris Stewart, Bastien
>> Teinturier, and others for input on the desired interface for
>> contracting protocols.
>>
>> Looking forward to hearing your thoughts!
>>
>> Best,
>> Gloria
>>
>> [0]:
>> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-January/019817.html
>> [1]:
>> https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-April/002639.html
>> [2]:
>> https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-June/002758.html
>> [3]:
>> https://github.com/t-bast/lightning-docs/blob/master/pinning-attacks.md
>> [4]:
>> https://github.com/revault/practical-revault/blob/master/transactions.md#cancel_tx
>> [5]:
>> https://github.com/discreetlogcontracts/dlcspecs/blob/master/Transactions.md#refund-transaction
>> [6]: https://gist.github.com/instagibbs/60264606e181451e977e439a49f69fe1
>> [7]:
>> https://btctranscripts.com/adopting-bitcoin/2021/2021-11-16-gloria-zhao-transaction-relay-policy/#lightning-attacks
>> [8]: https://youtu.be/fbWSQvJjKFs?t=1438
>> [9]:
>> https://www.reddit.com/r/Bitcoin/comments/unew4e/looks_like_70_mvb_of_transactions_just_got_dumped/
>> [10]: https://github.com/bitcoin/bitcoin/pull/7594
>> [11]: https://github.com/bitcoin/bitcoin/pull/7600
>> [12]: https://github.com/bitcoin/bitcoin/pull/6455#issuecomment-122716820
>> [13]: https://gist.github.com/sdaftuar/8756699bfcad4d3806ba9f3396d4e66a
>> [14]: https://github.com/bitcoin/bitcoin/issues/14895
>> [15]: https://github.com/bitcoin/bitcoin/pull/16401
>> [16]: https://github.com/bitcoin/bitcoin/pull/19621
>> [17]:
>> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-September/019464.html
>> [18]: https://github.com/users/glozow/projects/5/views/4?layout=board
>> [19]: https://gist.github.com/glozow/9b321cd3ef6505135c763112033ff2a7
>> [20]:
>> https://docs.google.com/presentation/d/1B__KlZO1VzxJGx-0DYChlWawaEmGJ9EGApEzrHqZpQc/edit?usp=sharing
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists•linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>

[-- Attachment #2: Type: text/html, Size: 38769 bytes --]

 permalink raw reply	[|nested]
2022-05-17 16:01 Gloria Zhao
  2022-05-17 17:56 ` Greg Sanders

  2022-05-18 18:40   ` Gloria Zhao
  2022-06-17 20:08 ` Antoine Riard
  
From: Anthony Towns @ 2022-05-18  0:35 UTC (permalink / raw)
  To: Gloria Zhao, Bitcoin Protocol Discussion

On Tue, May 17, 2022 at 12:01:04PM -0400, Gloria Zhao via bitcoin-dev wrote:
> ====New Messages====
> Three new protocol messages are added for use in any version of
> package relay. Additionally, each version of package relay must define
> its own inv type and "pckginfo" message version, referred to in this
> document as "MSG_PCKG" and "pckginfo" respectively. See
> BIP-v1-packages for a concrete example.

The "PCKG" abbreviation threw me for a loop; isn't the usual
abbreviation "PKG" ?

> =====sendpackages=====
> |version || uint32_t || 4 || Denotes a package version supported by the
> node.
> |max_count || uint32_t || 4 ||Specifies the maximum number of transactions
> per package this node is
> willing to accept.
> |max_weight || uint32_t || 4 ||Specifies the maximum total weight per
> package this node is willing
> to accept.

Does it make sense for these to be configurable, rather than implied
by the version? 

I presume the idea is to cope with people specifying different values for
-limitancestorcount or -limitancestorsize, but if people are regularly
relaying packages around, it seems like it becomes hard to have those
values really be configurable while being compatible with that?

I guess I'm asking: would it be better to either just not do sendpackages
at all if you're limiting ancestors in the mempool incompatibly; or
alternatively, would it be better to do the package relay, then reject
the particular package if it turns out too big, and log that you've
dropped it so that the node operator has some way of realising "whoops,
I'm not relaying packages properly because of how I configured my node"?

> 5. If 'fRelay==false' in a peer's version message, the node must not
>    send "sendpackages" to them. If a "sendpackages" message is
> received by a peer after sending `fRelay==false` in their version
> message, the sender should be disconnected.

Seems better to just say "if you set fRelay=false in your version
message, you must not send sendpackages"? You already won't do packages
with the peer if they don't also announce sendpackages.

> 7. If both peers send "wtxidrelay" and "sendpackages" with the same
>    version, the peers should announce, request, and send package
> information to each other.

Maybe: "You must not send sendpackages unless you also send wtxidrelay" ?


As I understand it, the two cases for the protocol flow are "I received
an orphan, and I'd like its ancestors please" which seems simple enough,
and "here's a child you may be interested in, even though you possibly
weren't interested in the parents of that child". I think the logic for
the latter is:

 * if tx C's fee rate is less than the peer's feefilter, skip it
   (will maybe treat it as a parent in some package later though)
 * if tx C's ancestor fee rate is less than the peer's feefilter, skip
   it?
 * look at the lowest ancestor fee rate for any of C's in-mempool
   parents
 * if that is higher than the peer's fee filter, send a normal INV
 * if it's lower than the peer's fee filter, send a PCKG INV

Are "getpckgtxns" / "pcktxns" really limited to packages, or are they
just a general way to request a batch of transactions? Particularly in
the case of requesting the parents of an orphan tx you already have,
it seems hard for the node receiving getpckgtxns to validate that the
txs are related in some way; but also it doesn't seem very necessary?

Maybe call those messages "getbatchtxns" and "batchtxns" and allow them to
be used more generally, potentially in ways unrelated to packages/cpfp?
The "only be sent if both peers agreed to do package relay" rule could
simply be dropped, I think.

> 4. The reciever uses the package information to decide how to request
>    the transactions. For example, if the receiver already has some of
> the transactions in their mempool, they only request the missing ones.
> They could also decide not to request the package at all based on the
> fee information provided.

Shouldn't the sender only be sending package announcements when they know
the recipient will be interested in the package, based on their feefilter?

> =====pckginfo1=====
> {|
> |  Field Name  ||  Type  ||  Size  ||   Purpose
> |-
> |blockhash || uint256 || 32 || The chain tip at which this package is
> defined.
> |-
> |pckg_fee||CAmount||4|| The sum total fees paid by all transactions in the
> package.

CAmount in consensus/amount.h is a int64_t so shouldn't this be 8
bytes? If you limit a package to 101kvB, an int32_t is enough to cover
any package with a fee rate of about 212 BTC/block or lower, though.

> |pckg_weight||int64_t||8|| The sum total weight of all transactions in the
> package.

The maximum block weight is 4M, and the default -limitancestorsize
presumably implies a max package weight of 404k; seems odd to provide
a uint64_t rather than an int32_t here, which easily allows either of
those values?

> 2. ''Only 1 child with unconfirmed parents.'' The package must consist
>    of one transaction and its unconfirmed parents. There must not be
> any other transactions in the package. Other dependency relationships
> may exist within the package (e.g. one parent may spend the output of
> another parent) provided that topological order is respected.

I think this means that some of the parents could also have unconfirmed
parents, but they won't be included in the package, and must be requested
via the recipient-initiated approach?

> 5. ''Total fees and weight.'' The 'total_fee' and 'total_weight'
>    fields must accurately represent the sum total of all transactions'
>    fees and weights as defined in BIP141, respectively.

Presumably this excludes any unconfirmed grandparents and earlier
ancestors since they aren't part of the package, in this approach? Doesn't
that make this both harder to calculate (assuming we already have
ancestor summaries) and less useful, in the case where those ancestors
have a lower fee rate?

> ''Q: Can "getpckgtxns" and "pckgtxns" messages contain only one
> transaction?''
> Yes.

This would be normal if you're requesting a single missing parent for
an orphan you've received, I think?

I'm slightly surprised the process is:

 ->  INV PCKG1 C
  <- GETDATA PCKG1 C
 ->  PCKGINFO1 blockhash A B C fee weight

rather than announcing the package fee info in the first message.
But if the sender is already applying the feefilter to the package before
announcing it, it probably doesn't matter, and means you're only getting
a 32B INV from every peer, rather than a 32*(n+2) PCKGINFO1 message from
every peer.

I guess tx relay is low priority enough that it wouldn't be worth tagging
some peers as "high bandwidth" and having them immediately announce the
PCKGINFO1 message, and skip the INV/GETDATA step?

Cheers,
aj



 permalink raw reply	[|nested]
2022-05-18  0:35 ` Anthony Towns

  2022-05-23 21:34     ` Anthony Towns
  
From: Gloria Zhao @ 2022-05-18 18:40 UTC (permalink / raw)
  To: Anthony Towns;  Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 14135 bytes --]

(To everyone):
I should have made it much clearer that version 1 is only supposed to solve
1 of the 2 use cases. I was a lot more focused on the fee-bumping use case,
since it’s more important. Orphan-fetching was added to the motivation
section last-minute because John Newbery mentioned to me “hey you could
deal with orphans really easily with this.” Of course,
child-with-unconfirmed-parents packages aren’t very useful for
orphan-fetching since non-parent ancestors are quite common.

Maybe a version 2 package for orphan-fetching could look like this:

“pckginfo2” message contains a tx with all of its ancestors

“MSG_PCKG2” inv type refers to a “pckginfo2” for a tx. You don’t send
inv(MSG_PCKG2), but a node can request getdata(MSG_PCKG2) for a transaction
they want the ancestors for, provided they sent sendpackages(version=2)
ahead of time. It seems to me that orphan-fetching only ever needs to be
receiver-initiated.

Protocol flow would look like this:
https://user-images.githubusercontent.com/25183001/168891185-1630f583-de47-4937-86b1-2652cf8852f2.png

We don’t have a policy for dealing with anything more than a child with its
direct parents, but I also don’t think anybody is relying on fee-bumping
more than 2 generations, so the validation logic here could probably just
submit them all individually. Maybe they can request a pckginfo1 if they
see something that’s too-low-fee, and/or use the
child-with-unconfirmed-parents logic opportunistically.

Thanks aj for the feedback! Responding:

> The "PCKG" abbreviation threw me for a loop; isn't the usual
> abbreviation "PKG" ?

Oh I didn't know that. I could change it if people feel strongly.

> Does it make sense for these to be configurable, rather than implied
> by the version?
> … would it be better to either just not do sendpackages
> at all if you're limiting ancestors in the mempool incompatibly

Effectively: if you’re setting your ancestor/descendant limits lower than
the default, you can’t do package relay. I wonder if this might be
controversial, since it adds pressure to adhere to Bitcoin Core’s current
mempool policy? I would be happy to do it this way, though - makes things
easier to implement.

> > 5. If 'fRelay==false' in a peer's version message, the node must not
> >    send "sendpackages" to them. If a "sendpackages" message is
> > received by a peer after sending `fRelay==false` in their version
> > message, the sender should be disconnected.

> Seems better to just say "if you set fRelay=false in your version
> message, you must not send sendpackages"? You already won't do packages
> with the peer if they don't also announce sendpackages.

I guess, theoretically, if you allow bloom filters with this peer, it’s
plausible they’re saying “fRelay=false, I’ll send you a bloom filter later,
and I’ll also want to talk about packages.”
I don’t know if that’s a use case we want to support - my gut reaction is
no.

> Maybe: "You must not send sendpackages unless you also send wtxidrelay" ?

Do you mean if we get a verack, and the peer sent “sendpackages” but not
“wtxidrelay,” we should disconnect them?

> As I understand it, the two cases for the protocol flow are "I received
> an orphan, and I'd like its ancestors please" which seems simple enough,
> and "here's a child you may be interested in, even though you possibly
> weren't interested in the parents of that child".

(Btw, please see my notes at the top of this email about separating those
two use cases. sorry for the confusion).

> I think the logic for the latter is: […]

I have it as: we send a PCKG INV when this transaction’s feerate is above
the fee filter, but one or more of its parents don’t. I don’t think using
ancestor feerate is better.
See this counterexample:
https://raw.githubusercontent.com/glozow/bitcoin-notes/master/mempool_garden/abc_1parent_2kids.png
A (0fee) has 2 kids, B (3sat/vB) and C (20sat/vB), everything’s the same
vsize. Let’s say the fee filter is 3sat/vB.
If we do it based on ancestor feerate, we won’t send B. But B is actually
fine; C is paying for A.

> Are "getpckgtxns" / "pcktxns" really limited to packages, or are they
> just a general way to request a batch of transactions?

> Maybe call those messages "getbatchtxns" and "batchtxns" and allow them to
> be used more generally, potentially in ways unrelated to packages/cpfp?

Indeed, it’s a general way to request a batch of transactions. I’ll
highlight that it is “all or nothing,” i.e. if the sender is missing any of
them, they’ll just send a notfound.
The idea here was to avoid downloading any transactions that can’t be
validated right away. With packages, this makes sense, because there are
dependency relationships. But if you’re requesting multiple unrelated
transactions, for example, it’s unnecessary. You might end up with even
more transaction data that’s just sitting around waiting to be validated.

> The "only be sent if both peers agreed to do package relay" rule could
> simply be dropped, I think.

Wouldn’t we need some way of saying “hey I support batchtxns?” Otherwise
you would have to guess by sending a request and waiting to see if it’s
ignored?

> Shouldn't the sender only be sending package announcements when they know
> the recipient will be interested in the package, based on their feefilter?

I think there are cases where the sender doesn’t necessarily know.
Consider this example:
https://raw.githubusercontent.com/glozow/bitcoin-notes/master/mempool_garden/rich_parent_bad_cpfp.png
D (5sat/vB) has 2 parents, A (0sat/vB) and B (20sat/vB). All same size.
Feefilter is 3sat/vB.
If the receiver already has B, they’ll know they can just reject the
package already based on the pckginfo.
But the sender doesn’t really know that. The sender just knows A is below
feerate and D is above. D is above the fee filter, and its ancestor feerate
is above the fee filter.

> CAmount in consensus/amount.h is a int64_t
> The maximum block weight is 4M…

 Oops yes. I think we just usually use int64_t for vsizes afaik. Agree that
it should be 8 bytes for fee, and 4 bytes is enough for vsize.

> I guess tx relay is low priority enough that it wouldn't be worth tagging
> some peers as "high bandwidth" and having them immediately announce the
> PCKGINFO1 message, and skip the INV/GETDATA step?

I had the same idea as well, but seemed unnecessary. It would reduce the
number of round trips, but I don’t think an extra round trip is that big of
a deal for transaction relay. Block relay, yes of course, but I don’t think
we care that much if it takes an extra second to send a transaction?

Best,
Gloria

On Tue, May 17, 2022 at 8:35 PM Anthony Towns <aj@erisian•com.au> wrote:

> On Tue, May 17, 2022 at 12:01:04PM -0400, Gloria Zhao via bitcoin-dev
> wrote:
> > ====New Messages====
> > Three new protocol messages are added for use in any version of
> > package relay. Additionally, each version of package relay must define
> > its own inv type and "pckginfo" message version, referred to in this
> > document as "MSG_PCKG" and "pckginfo" respectively. See
> > BIP-v1-packages for a concrete example.
>
> The "PCKG" abbreviation threw me for a loop; isn't the usual
> abbreviation "PKG" ?
>
> > =====sendpackages=====
> > |version || uint32_t || 4 || Denotes a package version supported by the
> > node.
> > |max_count || uint32_t || 4 ||Specifies the maximum number of
> transactions
> > per package this node is
> > willing to accept.
> > |max_weight || uint32_t || 4 ||Specifies the maximum total weight per
> > package this node is willing
> > to accept.
>
> Does it make sense for these to be configurable, rather than implied
> by the version?
>
> I presume the idea is to cope with people specifying different values for
> -limitancestorcount or -limitancestorsize, but if people are regularly
> relaying packages around, it seems like it becomes hard to have those
> values really be configurable while being compatible with that?
>
> I guess I'm asking: would it be better to either just not do sendpackages
> at all if you're limiting ancestors in the mempool incompatibly; or
> alternatively, would it be better to do the package relay, then reject
> the particular package if it turns out too big, and log that you've
> dropped it so that the node operator has some way of realising "whoops,
> I'm not relaying packages properly because of how I configured my node"?
>
> > 5. If 'fRelay==false' in a peer's version message, the node must not
> >    send "sendpackages" to them. If a "sendpackages" message is
> > received by a peer after sending `fRelay==false` in their version
> > message, the sender should be disconnected.
>
> Seems better to just say "if you set fRelay=false in your version
> message, you must not send sendpackages"? You already won't do packages
> with the peer if they don't also announce sendpackages.
>
> > 7. If both peers send "wtxidrelay" and "sendpackages" with the same
> >    version, the peers should announce, request, and send package
> > information to each other.
>
> Maybe: "You must not send sendpackages unless you also send wtxidrelay" ?
>
>
> As I understand it, the two cases for the protocol flow are "I received
> an orphan, and I'd like its ancestors please" which seems simple enough,
> and "here's a child you may be interested in, even though you possibly
> weren't interested in the parents of that child". I think the logic for
> the latter is:
>
>  * if tx C's fee rate is less than the peer's feefilter, skip it
>    (will maybe treat it as a parent in some package later though)
>  * if tx C's ancestor fee rate is less than the peer's feefilter, skip
>    it?
>  * look at the lowest ancestor fee rate for any of C's in-mempool
>    parents
>  * if that is higher than the peer's fee filter, send a normal INV
>  * if it's lower than the peer's fee filter, send a PCKG INV
>
> Are "getpckgtxns" / "pcktxns" really limited to packages, or are they
> just a general way to request a batch of transactions? Particularly in
> the case of requesting the parents of an orphan tx you already have,
> it seems hard for the node receiving getpckgtxns to validate that the
> txs are related in some way; but also it doesn't seem very necessary?
>
> Maybe call those messages "getbatchtxns" and "batchtxns" and allow them to
> be used more generally, potentially in ways unrelated to packages/cpfp?
> The "only be sent if both peers agreed to do package relay" rule could
> simply be dropped, I think.
>
> > 4. The reciever uses the package information to decide how to request
> >    the transactions. For example, if the receiver already has some of
> > the transactions in their mempool, they only request the missing ones.
> > They could also decide not to request the package at all based on the
> > fee information provided.
>
> Shouldn't the sender only be sending package announcements when they know
> the recipient will be interested in the package, based on their feefilter?
>
> > =====pckginfo1=====
> > {|
> > |  Field Name  ||  Type  ||  Size  ||   Purpose
> > |-
> > |blockhash || uint256 || 32 || The chain tip at which this package is
> > defined.
> > |-
> > |pckg_fee||CAmount||4|| The sum total fees paid by all transactions in
> the
> > package.
>
> CAmount in consensus/amount.h is a int64_t so shouldn't this be 8
> bytes? If you limit a package to 101kvB, an int32_t is enough to cover
> any package with a fee rate of about 212 BTC/block or lower, though.
>
> > |pckg_weight||int64_t||8|| The sum total weight of all transactions in
> the
> > package.
>
> The maximum block weight is 4M, and the default -limitancestorsize
> presumably implies a max package weight of 404k; seems odd to provide
> a uint64_t rather than an int32_t here, which easily allows either of
> those values?
>
> > 2. ''Only 1 child with unconfirmed parents.'' The package must consist
> >    of one transaction and its unconfirmed parents. There must not be
> > any other transactions in the package. Other dependency relationships
> > may exist within the package (e.g. one parent may spend the output of
> > another parent) provided that topological order is respected.
>
> I think this means that some of the parents could also have unconfirmed
> parents, but they won't be included in the package, and must be requested
> via the recipient-initiated approach?
>
> > 5. ''Total fees and weight.'' The 'total_fee' and 'total_weight'
> >    fields must accurately represent the sum total of all transactions'
> >    fees and weights as defined in BIP141, respectively.
>
> Presumably this excludes any unconfirmed grandparents and earlier
> ancestors since they aren't part of the package, in this approach? Doesn't
> that make this both harder to calculate (assuming we already have
> ancestor summaries) and less useful, in the case where those ancestors
> have a lower fee rate?
>
> > ''Q: Can "getpckgtxns" and "pckgtxns" messages contain only one
> > transaction?''
> > Yes.
>
> This would be normal if you're requesting a single missing parent for
> an orphan you've received, I think?
>
> I'm slightly surprised the process is:
>
>  ->  INV PCKG1 C
>   <- GETDATA PCKG1 C
>  ->  PCKGINFO1 blockhash A B C fee weight
>
> rather than announcing the package fee info in the first message.
> But if the sender is already applying the feefilter to the package before
> announcing it, it probably doesn't matter, and means you're only getting
> a 32B INV from every peer, rather than a 32*(n+2) PCKGINFO1 message from
> every peer.
>
> I guess tx relay is low priority enough that it wouldn't be worth tagging
> some peers as "high bandwidth" and having them immediately announce the
> PCKGINFO1 message, and skip the INV/GETDATA step?
>
> Cheers,
> aj
>
>

[-- Attachment #2: Type: text/html, Size: 16153 bytes --]

 permalink raw reply	[|nested]
2022-05-18 18:40   ` Gloria Zhao

  2022-05-24  1:13       ` Gloria Zhao
  
From: Anthony Towns @ 2022-05-23 21:34 UTC (permalink / raw)
  To: bitcoin-dev

On Wed, May 18, 2022 at 02:40:58PM -0400, Gloria Zhao via bitcoin-dev wrote:
> > Does it make sense for these to be configurable, rather than implied
> > by the version?
> > … would it be better to either just not do sendpackages
> > at all if you're limiting ancestors in the mempool incompatibly
> Effectively: if you’re setting your ancestor/descendant limits lower than
> the default, you can’t do package relay. I wonder if this might be
> controversial, since it adds pressure to adhere to Bitcoin Core’s current
> mempool policy? I would be happy to do it this way, though - makes things
> easier to implement.

How about looking at it the other way: if you're writing a protocol that's
dependent on people seeing that a package as a whole pays a competitive
feerate, don't you want to know in advance what conditions the network
is going to impose on your transactions in order to consider them as a
package? In that case, aren't the "depth" and "size" constraints things
we should specify in a standard?

(The above's not a rhetorical question; I'm not sure what the answer is.
And even if it's "yes", maybe core's defaults should be reconsidered
rather than standardised as-is)

Worst case, you could presumably do a new package relay version with
different constraints, if needed.

> > > 5. If 'fRelay==false' in a peer's version message, the node must not
> > >    send "sendpackages" to them. If a "sendpackages" message is
> > > received by a peer after sending `fRelay==false` in their version
> > > message, the sender should be disconnected.
> > Seems better to just say "if you set fRelay=false in your version
> > message, you must not send sendpackages"? You already won't do packages
> > with the peer if they don't also announce sendpackages.
> I guess, theoretically, if you allow bloom filters with this peer, it’s
> plausible they’re saying “fRelay=false, I’ll send you a bloom filter later,
> and I’ll also want to talk about packages.”

I was just meaning "it's okay to send VERSION fRelay=true then immediately
send WTXIDRELAY then immediately send SENDPACKAGES" without having to
first verify what the other guy's fRelay was set to. On the other hand,
you do already have to verify the other guy's version is high enough,
but it would be kind-of nice to move towards just announcing the features
you support, and not having to make it a multistep negotiation...

> > Maybe: "You must not send sendpackages unless you also send wtxidrelay" ?
> Do you mean if we get a verack, and the peer sent “sendpackages” but not
> “wtxidrelay,” we should disconnect them?

Yes.

> I have it as: we send a PCKG INV when this transaction’s feerate is above
> the fee filter, but one or more of its parents don’t. I don’t think using
> ancestor feerate is better.
> See this counterexample:
> https://raw.githubusercontent.com/glozow/bitcoin-notes/master/mempool_garden/abc_1parent_2kids.png
> A (0fee) has 2 kids, B (3sat/vB) and C (20sat/vB), everything’s the same
> vsize. Let’s say the fee filter is 3sat/vB.
> If we do it based on ancestor feerate, we won’t send B. But B is actually
> fine; C is paying for A.

But that only works if the receiver also has C, in which case they also
have A, and you don't need package relay to do anything with B? If they
didn't have C already, then relaying {A,B} would be a waste of time,
because {A,B} would be rejected as only paying 1.5sat/vB or whatever..

If you switch it to being:

  A (0 sats, 200vB)
  B (2000 sats, 200vB, spends A:0)
  C (200 sats, 200vB)
  D (1000 sats, 200vB, sepnds A:1, C:0)

then you get:

  A alone = 0s/vB
  B+A = 5s/vB

  C alone = 1s/vB
  D+C+A = 2s/vB
  D+C = 3s/vB      (B+A already at 5s/vB)

which I think recovers your point, while also having all the details
only be dealing with direct parents.

> > Are "getpckgtxns" / "pcktxns" really limited to packages, or are they
> > just a general way to request a batch of transactions?
> > Maybe call those messages "getbatchtxns" and "batchtxns" and allow them to
> > be used more generally, potentially in ways unrelated to packages/cpfp?
> Indeed, it’s a general way to request a batch of transactions. I’ll
> highlight that it is “all or nothing,” i.e. if the sender is missing any of
> them, they’ll just send a notfound.
> The idea here was to avoid downloading any transactions that can’t be
> validated right away.

Right; maybe I should just be calling a "batch of packages to be validated
together" a "tx package" in the first place.

Maybe it would be worth emphasising that you should be expecting to
validate all the txs you receive as a response to getpckgtxns (getpkgtxs
:) all at the same time, and immediately upon receiving them?

> > The "only be sent if both peers agreed to do package relay" rule could
> > simply be dropped, I think.
> Wouldn’t we need some way of saying “hey I support batchtxns?” Otherwise
> you would have to guess by sending a request and waiting to see if it’s
> ignored?

Sure, perhaps I should have said leave that rule, but drop the following
"should be disconnected" rule, so that other BIPs could add in other
ways of negotiating the connection in future? *shrug*

> > Shouldn't the sender only be sending package announcements when they know
> > the recipient will be interested in the package, based on their feefilter?
> I think there are cases where the sender doesn’t necessarily know.
> Consider this example:
> https://raw.githubusercontent.com/glozow/bitcoin-notes/master/mempool_garden/rich_parent_bad_cpfp.png
> D (5sat/vB) has 2 parents, A (0sat/vB) and B (20sat/vB). All same size.
> Feefilter is 3sat/vB.
> If the receiver already has B, they’ll know they can just reject the
> package already based on the pckginfo.
> But the sender doesn’t really know that. The sender just knows A is below
> feerate and D is above. D is above the fee filter, and its ancestor feerate
> is above the fee filter.

The sender would also need to know whether or not there's some other
child E that pays for A sufficiently?

If you're asking for the package for "D", would a response telling you:

  txid_D (500 sat, 100vB)
  txid_A (0 sat, 100vB)
  txid_B (2000 sat, 100 vB)

be better, in that case? Then the receiver can maybe do the logic
themselves to figure out that they already have A in their mempool
so it's fine, or not?

If you've got a package for X, and its direct parents P1..Pn, then
I think the logic would be:

  * is X alone above my fee rate? no, then forget it
  * otherwise, s := X.size, f := X.fees, R := [X]
  * for P = P1..Pn:
    * do I already have P? then skip to the next parent
    * s += P.size, f += P.fees, R += [P]
  * if f/s above my fee rate floor? if so, request all the txs in R

and you'd request txs if-and-only-if they're a match for you mempool rate?

If you have a tx with 20 in-mempool parents, then the pkginfo1 message
as proposed would be 737 bytes; including all the fee/size info would be
957 bytes, maybe a 30% increase. Might be worth it though?

Cheers,
aj



 permalink raw reply	[|nested]
2022-05-23 21:34     ` Anthony Towns

  2022-05-24 19:48         ` Anthony Towns
  
From: Gloria Zhao @ 2022-05-24  1:13 UTC (permalink / raw)
  To: Anthony Towns, Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 10886 bytes --]

Hi aj,

> if you're writing a protocol that's
> dependent on people seeing that a package as a whole pays a competitive
> feerate, don't you want to know in advance what conditions the network
> is going to impose on your transactions in order to consider them as a
> package?

I do think unifying the size/count constraints would result in a more
stable/easier to reason about interface for L2 devs. Then the requirement
for propagation is just a path of nodes that support v1 package relay, and
it’s implied their mempool policy supports it as well. Also seems like it
could be a fingerprinting problem for nodes to give very specific
count/size limits.

> (… maybe core's defaults should be reconsidered rather than standardised
as-is)

> Worst case, you could presumably do a new package relay version with
> different constraints, if needed.

Maybe this was my actual concern. I think the defaults are safe but it’s
not like they’ve been proven to be optimal. This creates an obstacle to
changing them, especially if we want to make them smaller. But I think it’s
unlikely we’ll do that, and adding another version for new constraints
doesn’t seem too bad.


(Agreed with everything here, thanks for the feedback and clarifications!)
TLDR, making these changes:
- Count and size are implied by the version. Version 1 is specifically
child-with-unconfirmed-parents, where the whole package is at most 25
transactions and 101KvB.
- Announce sendpackages based on our own state. It’s ok to send
“sendpackages” if they sent fRelay=false.
- At verack, require fRelay=true and wtxidrelay if they sent sendpackages,
otherwise disconnect.
- If we get “getpckgtxns” or “pckgtxns” without having negotiated
“sendpackages” ahead of time, ignore, don’t disconnect. Emphasize that the
intention is to validate all of the transactions received through
“pckgtxns” together.

> If you're asking for the package for "D", would a response telling you:
>   txid_D (500 sat, 100vB)
>   txid_A (0 sat, 100vB)
>   txid_B (2000 sat, 100 vB)
> be better, in that case? Then the receiver can maybe do the logic
> themselves to figure out that they already have A in their mempool
> so it's fine, or not?

Right, I also considered giving the fees and sizes of each transaction in
the package in “pckginfo1”. But I don’t think that information provides
additional meaning unless you know the exact topology, i.e. also know if
the parents have dependency relationships between them. For instance, in
the {A, B, D} package there, even if you have the information listed, your
decision should be different depending on whether B spends from A. The only
thing you know for sure about a child with direct parents is: if the
aggregate feerate is too low, you won’t want the child since it depends on
everyone else. If there’s a good-feerate transaction in there that doesn’t
have a dependency, you’re fine as long as someone sends it to you
individually.

Best,
Gloria

On Mon, May 23, 2022 at 2:34 PM Anthony Towns via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> wrote:

> On Wed, May 18, 2022 at 02:40:58PM -0400, Gloria Zhao via bitcoin-dev
> wrote:
> > > Does it make sense for these to be configurable, rather than implied
> > > by the version?
> > > … would it be better to either just not do sendpackages
> > > at all if you're limiting ancestors in the mempool incompatibly
> > Effectively: if you’re setting your ancestor/descendant limits lower than
> > the default, you can’t do package relay. I wonder if this might be
> > controversial, since it adds pressure to adhere to Bitcoin Core’s current
> > mempool policy? I would be happy to do it this way, though - makes things
> > easier to implement.
>
> How about looking at it the other way: if you're writing a protocol that's
> dependent on people seeing that a package as a whole pays a competitive
> feerate, don't you want to know in advance what conditions the network
> is going to impose on your transactions in order to consider them as a
> package? In that case, aren't the "depth" and "size" constraints things
> we should specify in a standard?
>
> (The above's not a rhetorical question; I'm not sure what the answer is.
> And even if it's "yes", maybe core's defaults should be reconsidered
> rather than standardised as-is)
>
> Worst case, you could presumably do a new package relay version with
> different constraints, if needed.
>
> > > > 5. If 'fRelay==false' in a peer's version message, the node must not
> > > >    send "sendpackages" to them. If a "sendpackages" message is
> > > > received by a peer after sending `fRelay==false` in their version
> > > > message, the sender should be disconnected.
> > > Seems better to just say "if you set fRelay=false in your version
> > > message, you must not send sendpackages"? You already won't do packages
> > > with the peer if they don't also announce sendpackages.
> > I guess, theoretically, if you allow bloom filters with this peer, it’s
> > plausible they’re saying “fRelay=false, I’ll send you a bloom filter
> later,
> > and I’ll also want to talk about packages.”
>
> I was just meaning "it's okay to send VERSION fRelay=true then immediately
> send WTXIDRELAY then immediately send SENDPACKAGES" without having to
> first verify what the other guy's fRelay was set to. On the other hand,
> you do already have to verify the other guy's version is high enough,
> but it would be kind-of nice to move towards just announcing the features
> you support, and not having to make it a multistep negotiation...
>
> > > Maybe: "You must not send sendpackages unless you also send
> wtxidrelay" ?
> > Do you mean if we get a verack, and the peer sent “sendpackages” but not
> > “wtxidrelay,” we should disconnect them?
>
> Yes.
>
> > I have it as: we send a PCKG INV when this transaction’s feerate is above
> > the fee filter, but one or more of its parents don’t. I don’t think using
> > ancestor feerate is better.
> > See this counterexample:
> >
> https://raw.githubusercontent.com/glozow/bitcoin-notes/master/mempool_garden/abc_1parent_2kids.png
> > A (0fee) has 2 kids, B (3sat/vB) and C (20sat/vB), everything’s the same
> > vsize. Let’s say the fee filter is 3sat/vB.
> > If we do it based on ancestor feerate, we won’t send B. But B is actually
> > fine; C is paying for A.
>
> But that only works if the receiver also has C, in which case they also
> have A, and you don't need package relay to do anything with B? If they
> didn't have C already, then relaying {A,B} would be a waste of time,
> because {A,B} would be rejected as only paying 1.5sat/vB or whatever..
>
> If you switch it to being:
>
>   A (0 sats, 200vB)
>   B (2000 sats, 200vB, spends A:0)
>   C (200 sats, 200vB)
>   D (1000 sats, 200vB, sepnds A:1, C:0)
>
> then you get:
>
>   A alone = 0s/vB
>   B+A = 5s/vB
>
>   C alone = 1s/vB
>   D+C+A = 2s/vB
>   D+C = 3s/vB      (B+A already at 5s/vB)
>
> which I think recovers your point, while also having all the details
> only be dealing with direct parents.
>
> > > Are "getpckgtxns" / "pcktxns" really limited to packages, or are they
> > > just a general way to request a batch of transactions?
> > > Maybe call those messages "getbatchtxns" and "batchtxns" and allow
> them to
> > > be used more generally, potentially in ways unrelated to packages/cpfp?
> > Indeed, it’s a general way to request a batch of transactions. I’ll
> > highlight that it is “all or nothing,” i.e. if the sender is missing any
> of
> > them, they’ll just send a notfound.
> > The idea here was to avoid downloading any transactions that can’t be
> > validated right away.
>
> Right; maybe I should just be calling a "batch of packages to be validated
> together" a "tx package" in the first place.
>
> Maybe it would be worth emphasising that you should be expecting to
> validate all the txs you receive as a response to getpckgtxns (getpkgtxs
> :) all at the same time, and immediately upon receiving them?
>
> > > The "only be sent if both peers agreed to do package relay" rule could
> > > simply be dropped, I think.
> > Wouldn’t we need some way of saying “hey I support batchtxns?” Otherwise
> > you would have to guess by sending a request and waiting to see if it’s
> > ignored?
>
> Sure, perhaps I should have said leave that rule, but drop the following
> "should be disconnected" rule, so that other BIPs could add in other
> ways of negotiating the connection in future? *shrug*
>
> > > Shouldn't the sender only be sending package announcements when they
> know
> > > the recipient will be interested in the package, based on their
> feefilter?
> > I think there are cases where the sender doesn’t necessarily know.
> > Consider this example:
> >
> https://raw.githubusercontent.com/glozow/bitcoin-notes/master/mempool_garden/rich_parent_bad_cpfp.png
> > D (5sat/vB) has 2 parents, A (0sat/vB) and B (20sat/vB). All same size.
> > Feefilter is 3sat/vB.
> > If the receiver already has B, they’ll know they can just reject the
> > package already based on the pckginfo.
> > But the sender doesn’t really know that. The sender just knows A is below
> > feerate and D is above. D is above the fee filter, and its ancestor
> feerate
> > is above the fee filter.
>
> The sender would also need to know whether or not there's some other
> child E that pays for A sufficiently?
>
> If you're asking for the package for "D", would a response telling you:
>
>   txid_D (500 sat, 100vB)
>   txid_A (0 sat, 100vB)
>   txid_B (2000 sat, 100 vB)
>
> be better, in that case? Then the receiver can maybe do the logic
> themselves to figure out that they already have A in their mempool
> so it's fine, or not?
>
> If you've got a package for X, and its direct parents P1..Pn, then
> I think the logic would be:
>
>   * is X alone above my fee rate? no, then forget it
>   * otherwise, s := X.size, f := X.fees, R := [X]
>   * for P = P1..Pn:
>     * do I already have P? then skip to the next parent
>     * s += P.size, f += P.fees, R += [P]
>   * if f/s above my fee rate floor? if so, request all the txs in R
>
> and you'd request txs if-and-only-if they're a match for you mempool rate?
>
> If you have a tx with 20 in-mempool parents, then the pkginfo1 message
> as proposed would be 737 bytes; including all the fee/size info would be
> 957 bytes, maybe a 30% increase. Might be worth it though?
>
> Cheers,
> aj
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 12855 bytes --]

 permalink raw reply	[|nested]
2022-05-24  1:13       ` Gloria Zhao

  2022-05-24 21:05           ` Gloria Zhao
  
From: Anthony Towns @ 2022-05-24 19:48 UTC (permalink / raw)
  To: Gloria Zhao, Bitcoin Protocol Discussion

On 23 May 2022 9:13:43 pm GMT-04:00, Gloria Zhao <gloriajzhao@gmail•com> wrote:
>> If you're asking for the package for "D", would a response telling you:
>>   txid_D (500 sat, 100vB)
>>   txid_A (0 sat, 100vB)
>>   txid_B (2000 sat, 100 vB)
>> be better, in that case? Then the receiver can maybe do the logic
>> themselves to figure out that they already have A in their mempool
>> so it's fine, or not?
>Right, I also considered giving the fees and sizes of each transaction in
>the package in “pckginfo1”. But I don’t think that information provides
>additional meaning unless you know the exact topology, i.e. also know if
>the parents have dependency relationships between them. For instance, in
>the {A, B, D} package there, even if you have the information listed, your
>decision should be different depending on whether B spends from A.

I don't think that's true? We already know D is above our fee floor so if B with A is also above the floor, we want them all, but also if B isn't above the floor, but all of them combined are, then we also do?

If you've got (A,B,C,X) where B spends A and X spends A,B,C where X+C is below fee floor while A+B and A+B+C+X are above fee floor you have the problem though.

Is it plausible to add the graph in?

Cheers,
aj



-- 
Sent from my phone.


 permalink raw reply	[|nested]
2022-05-24 19:48         ` Anthony Towns

  2022-05-24 23:43             ` Eric Voskuil
  2022-05-25 18:55             ` Anthony Towns
  
From: Gloria Zhao @ 2022-05-24 21:05 UTC (permalink / raw)
  To: Anthony Towns;  Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 3430 bytes --]

Hi aj,

> If you've got (A,B,C,X) where B spends A and X spends A,B,C where X+C is
below fee floor while A+B and A+B+C+X are above fee floor you have the
problem though.

To clarify, in this situation, I'm imagining something like
A: 0 sat, 100vB
B: 1500 sat, 100vB
C: 0 sat, 100vB
X: 500 sat, 100vB
feerate floor is 3sat/vB

With the algo:
>  * is X alone above my fee rate? no, then forget it
>  * otherwise, s := X.size, f := X.fees, R := [X]
>  * for P = P1..Pn:
>   * do I already have P? then skip to the next parent
>   * s += P.size, f += P.fees, R += [P]
>  * if f/s above my fee rate floor? if so, request all the txs in R

We'd erroneously ask for A+B+C+X, but really we should only take A+B.
But wouldn't A+B also be a package that was announced for B?
Please lmk if you were imagining something different. I think I may be
missing something.

> Is it plausible to add the graph in?

Fun to think about. Most basic design would be to represent {spends,
doesn’t spend} for a previous transaction in the package as a bit. Can
think of it as a matrix where row i, column j tells you whether Tx j
(directly) spends Tx i.
But of course you can omit the last row, since the child spends all of
them. And since topological ordering is a requirement, you only need as
many bits as there are transactions preceding this one in the package.
If you have up to 24 parents, you need 1 + 2 + ... + 23 bits to codify
spending for the 2nd ... 24th parent. For a maximum 25 transactions,
23*24/2 = 276, seems like 36 bytes for a child-with-parents package. A few
more for tx-with-ancestors.
Then you can split it up into sub-packages and everything. Still not sure
if we really need to.

Also side note, since there are no size/count params, wondering if we
should just have "version" in "sendpackages" be a bit field instead of
sending a message for each version. 32 versions should be enough right?

Best,
Gloria

On Tue, 24 May 2022 at 12:48 Anthony Towns <aj@erisian•com.au> wrote:

> On 23 May 2022 9:13:43 pm GMT-04:00, Gloria Zhao <gloriajzhao@gmail•com>
> wrote:
> >> If you're asking for the package for "D", would a response telling you:
> >>   txid_D (500 sat, 100vB)
> >>   txid_A (0 sat, 100vB)
> >>   txid_B (2000 sat, 100 vB)
> >> be better, in that case? Then the receiver can maybe do the logic
> >> themselves to figure out that they already have A in their mempool
> >> so it's fine, or not?
> >Right, I also considered giving the fees and sizes of each transaction in
> >the package in “pckginfo1”. But I don’t think that information provides
> >additional meaning unless you know the exact topology, i.e. also know if
> >the parents have dependency relationships between them. For instance, in
> >the {A, B, D} package there, even if you have the information listed, your
> >decision should be different depending on whether B spends from A.
>
> I don't think that's true? We already know D is above our fee floor so if
> B with A is also above the floor, we want them all, but also if B isn't
> above the floor, but all of them combined are, then we also do?
>
> If you've got (A,B,C,X) where B spends A and X spends A,B,C where X+C is
> below fee floor while A+B and A+B+C+X are above fee floor you have the
> problem though.
>
> Is it plausible to add the graph in?
>
> Cheers,
> aj
>
>
>
> --
> Sent from my phone.
>

[-- Attachment #2: Type: text/html, Size: 4690 bytes --]

 permalink raw reply	[|nested]
2022-05-24 21:05           ` Gloria Zhao

  2022-05-25 18:55             ` Anthony Towns
  
From: Eric Voskuil @ 2022-05-24 23:43 UTC (permalink / raw)
  To: Gloria Zhao, Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 5134 bytes --]

The set of txs is the graph. Anything else would just reproduce the tx graph which must be traversed in any case.

Similarly the set of txs is the fee, the sigops, the size, and the weight. The only information required by packaging is the association of the txs with each other for the purpose of aggregate (vs. individual) net reward consideration.

Since a package can only be reasonably considered for a single block, there is a natural effective limit on acceptable package size. Since any number of individual txs may be transmitted, and the size/weight/sigops of one tx is bounded only by block validity, there is no reason to put any other constraints on packages. A package is just a set of txs that may fit into a block and may collectively be worth mining. A rational package is just a block or compact block without the header. Making it any more than that is unnecessary complexity.

If parts of a package satisfy profitability constraints, they will be accepted/mined and if other parts do not, they will be rejected. There’s no preventing this.

The only pertinent feature missing in the p2p protocol is the ability to associate a set of txs for consideration, where the set (or subset) may satisfy profitability constraints that would not be satisfied if the txs were considered individually.

e

> On May 24, 2022, at 16:21, Gloria Zhao via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
> 
> ﻿
> Hi aj,
> 
> > If you've got (A,B,C,X) where B spends A and X spends A,B,C where X+C is below fee floor while A+B and A+B+C+X are above fee floor you have the problem though.
> 
> To clarify, in this situation, I'm imagining something like
> A: 0 sat, 100vB
> B: 1500 sat, 100vB
> C: 0 sat, 100vB
> X: 500 sat, 100vB
> feerate floor is 3sat/vB
> 
> With the algo:
> >  * is X alone above my fee rate? no, then forget it
> >  * otherwise, s := X.size, f := X.fees, R := [X]
> >  * for P = P1..Pn:
> >   * do I already have P? then skip to the next parent
> >   * s += P.size, f += P.fees, R += [P]
> >  * if f/s above my fee rate floor? if so, request all the txs in R
> 
> We'd erroneously ask for A+B+C+X, but really we should only take A+B.
> But wouldn't A+B also be a package that was announced for B?
> Please lmk if you were imagining something different. I think I may be missing something.
> 
> > Is it plausible to add the graph in?
> 
> Fun to think about. Most basic design would be to represent {spends, doesn’t spend} for a previous transaction in the package as a bit. Can think of it as a matrix where row i, column j tells you whether Tx j (directly) spends Tx i.
> But of course you can omit the last row, since the child spends all of them. And since topological ordering is a requirement, you only need as many bits as there are transactions preceding this one in the package.
> If you have up to 24 parents, you need 1 + 2 + ... + 23 bits to codify spending for the 2nd ... 24th parent. For a maximum 25 transactions, 23*24/2 = 276, seems like 36 bytes for a child-with-parents package. A few more for tx-with-ancestors.
> Then you can split it up into sub-packages and everything. Still not sure if we really need to.
> 
> Also side note, since there are no size/count params, wondering if we should just have "version" in "sendpackages" be a bit field instead of sending a message for each version. 32 versions should be enough right?
> 
> Best,
> Gloria
> 
>> On Tue, 24 May 2022 at 12:48 Anthony Towns <aj@erisian•com.au> wrote:
>> On 23 May 2022 9:13:43 pm GMT-04:00, Gloria Zhao <gloriajzhao@gmail•com> wrote:
>> >> If you're asking for the package for "D", would a response telling you:
>> >>   txid_D (500 sat, 100vB)
>> >>   txid_A (0 sat, 100vB)
>> >>   txid_B (2000 sat, 100 vB)
>> >> be better, in that case? Then the receiver can maybe do the logic
>> >> themselves to figure out that they already have A in their mempool
>> >> so it's fine, or not?
>> >Right, I also considered giving the fees and sizes of each transaction in
>> >the package in “pckginfo1”. But I don’t think that information provides
>> >additional meaning unless you know the exact topology, i.e. also know if
>> >the parents have dependency relationships between them. For instance, in
>> >the {A, B, D} package there, even if you have the information listed, your
>> >decision should be different depending on whether B spends from A.
>> 
>> I don't think that's true? We already know D is above our fee floor so if B with A is also above the floor, we want them all, but also if B isn't above the floor, but all of them combined are, then we also do?
>> 
>> If you've got (A,B,C,X) where B spends A and X spends A,B,C where X+C is below fee floor while A+B and A+B+C+X are above fee floor you have the problem though.
>> 
>> Is it plausible to add the graph in?
>> 
>> Cheers,
>> aj
>> 
>> 
>> 
>> -- 
>> Sent from my phone.
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[-- Attachment #2: Type: text/html, Size: 6844 bytes --]

 permalink raw reply	[|nested]
2022-05-24 21:05           ` Gloria Zhao
  2022-05-24 23:43             ` Eric Voskuil

  2022-05-25 20:52               ` eric
  2022-05-28  1:54               ` Gloria Zhao
  
From: Anthony Towns @ 2022-05-25 18:55 UTC (permalink / raw)
  To: Gloria Zhao, Bitcoin Protocol Discussion

On 24 May 2022 5:05:35 pm GMT-04:00, Gloria Zhao via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:
>To clarify, in this situation, I'm imagining something like
>A: 0 sat, 100vB
>B: 1500 sat, 100vB
>C: 0 sat, 100vB
>X: 500 sat, 100vB
>feerate floor is 3sat/vB
>
>With the algo:
>>  * is X alone above my fee rate? no, then forget it
>>  * otherwise, s := X.size, f := X.fees, R := [X]
>>  * for P = P1..Pn:
>>   * do I already have P? then skip to the next parent
>>   * s += P.size, f += P.fees, R += [P]
>>  * if f/s above my fee rate floor? if so, request all the txs in R
>
>We'd erroneously ask for A+B+C+X, but really we should only take A+B.
>But wouldn't A+B also be a package that was announced for B?

In theory, yes, but maybe it was announced earlier (while our node was down?) or had dropped from our mempool or similar, either way we don't have those txs yet.

>Please lmk if you were imagining something different. I think I may be
>missing something.

That's what I was thinking, yes.

So the other thing is what happens if the peer announcing packages to us is dishonest?

They announce pkg X, say X has parents A B C and the fee rate is garbage. But actually X has parent D and the fee rate is excellent. Do we request the package from another peer, or every peer, to double check? Otherwise we're allowing the first peer we ask about a package to censor that tx from us?

I think the fix for that is just to provide the fee and weight when announcing the package rather than only being asked for its info? Then if one peer makes it sound like a good deal you ask for the parent txids from them, dedupe, request, and verify they were honest about the parents.

>> Is it plausible to add the graph in?

Likewise, I think you'd have to have the graph info from many nodes if you're going to make decisions based on it and don't want hostile peers to be able to trick you into ignoring txs.

Other idea: what if you encode the parent txs as a short hash of the wtxid (something like bip152 short ids? perhaps seeded per peer so collisions will be different per peer?) and include that in the inv announcement? Would that work to avoid a round trip almost all of the time, while still giving you enough info to save bw by deduping parents?


> For a maximum 25 transactions,
>23*24/2 = 276, seems like 36 bytes for a child-with-parents package.

If you're doing short ids that's maybe 25*4B=100B already, then the above is up to 36% overhead, I guess. Might be worth thinking more about, but maybe more interesting with ancestors than just parents.

>Also side note, since there are no size/count params, wondering if we
>should just have "version" in "sendpackages" be a bit field instead of
>sending a message for each version. 32 versions should be enough right?

Maybe but a couple of messages per connection doesn't really seem worth arguing about?

Cheers,
aj


-- 
Sent from my phone.


 permalink raw reply	[|nested]
2022-05-25 18:55             ` Anthony Towns

  2022-05-26  2:59                 ` eric
  2022-05-28  1:54               ` Gloria Zhao
  
From: eric @ 2022-05-25 20:52 UTC (permalink / raw)
  To: 'Anthony Towns', 'Bitcoin Protocol Discussion',
	'Gloria Zhao'

> From: bitcoin-dev <bitcoin-dev-bounces@lists•linuxfoundation.org> On
Behalf
> Of Anthony Towns via bitcoin-dev
> Sent: Wednesday, May 25, 2022 11:56 AM

> So the other thing is what happens if the peer announcing packages to us
is
> dishonest?
> 
> They announce pkg X, say X has parents A B C and the fee rate is garbage.
But
> actually X has parent D and the fee rate is excellent. Do we request the
> package from another peer, or every peer, to double check? Otherwise we're
> allowing the first peer we ask about a package to censor that tx from us?
> 
> I think the fix for that is just to provide the fee and weight when
announcing
> the package rather than only being asked for its info? Then if one peer
makes
> it sound like a good deal you ask for the parent txids from them, dedupe,
> request, and verify they were honest about the parents.

Single tx broadcasts do not carry an advertised fee rate, however the'
feefilter' message (BIP133) provides this distinction. This should be
interpreted as applicable to packages. Given this message there is no reason
to send a (potentially bogus) fee rate with every package. It can only be
validated by obtaining the full set of txs, and the only recourse is
dropping (etc.) the peer, as is the case with single txs. Relying on the
existing message is simpler, more consistent, and more efficient.

> >> Is it plausible to add the graph in?
> 
> Likewise, I think you'd have to have the graph info from many nodes if
you're
> going to make decisions based on it and don't want hostile peers to be
able to
> trick you into ignoring txs.
> 
> Other idea: what if you encode the parent txs as a short hash of the wtxid
> (something like bip152 short ids? perhaps seeded per peer so collisions
will
> be different per peer?) and include that in the inv announcement? Would
> that work to avoid a round trip almost all of the time, while still giving
you
> enough info to save bw by deduping parents?

As I suggested earlier, a package is fundamentally a compact block (or
block) announcement without the header. Compact block (BIP152) announcement
is already well-defined and widely implemented. A node should never be
required to retain an orphan, and BIP152 ensures this is not required.

Once a validated set of txs within the package has been obtained with
sufficient fee, a fee-optimal node would accept the largest subgraph of the
package that conforms to fee constraints and drop any peer that provides a
package for which the full graph does not.

Let us not reinvent the wheel and/or introduce accidental complexity. I see
no reason why packaging is not simply BIP152 without the 'header' field, an
updated protocol version, and the following sort of changes to names:

sendpkg
MSG_CMPCT_PKG
cmpctpkg
getpkgtxn
pkgtxn

> > For a maximum 25 transactions,
> >23*24/2 = 276, seems like 36 bytes for a child-with-parents package.
> 
> If you're doing short ids that's maybe 25*4B=100B already, then the above
is
> up to 36% overhead, I guess. Might be worth thinking more about, but maybe
> more interesting with ancestors than just parents.
> 
> >Also side note, since there are no size/count params,

Size is restricted in the same manner as block and transaction broadcasts,
by consensus. If the fee rate is sufficient there would be no reason to
preclude any valid size up to what can be mined in one block (packaging
across blocks is not economically rational under the assumption that one
miner cannot expect to mine multiple blocks in a row). Count is incorporated
into BIP152 as 'shortids_length'.

> > wondering if we
> >should just have "version" in "sendpackages" be a bit field instead of
> >sending a message for each version. 32 versions should be enough right?

Adding versioning to individual protocols is just a reflection of the
insufficiency of the initial protocol versioning design, and that of the
various ad-hoc changes to it (including yet another approach in this
proposal) that have been introduced to compensate for it, though I'll
address this in an independent post at some point.

Best,
e

> Maybe but a couple of messages per connection doesn't really seem worth
> arguing about?
> 
> Cheers,
> aj
> 
> 
> --
> Sent from my phone.
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev



 permalink raw reply	[|nested]
2022-05-25 20:52               ` eric

  2022-06-07 17:44                   ` Gloria Zhao
  
From: eric @ 2022-05-26  2:59 UTC (permalink / raw)
  To: 'Anthony Towns', 'Bitcoin Protocol Discussion',
	'Gloria Zhao'

Given that packages have no header, the package requires identity in a
BIP152 scheme. For example 'header' and 'blockhash' fields can be replaced
with a Merkle root (e.g. "identity" field) for the package, uniquely
identifying the partially-ordered set of txs. And use of 'getdata' (to
obtain a package by hash) can be eliminated (not a use case).

e

> -----Original Message-----
> From: eric@voskuil•org <eric@voskuil•org>
> Sent: Wednesday, May 25, 2022 1:52 PM
> To: 'Anthony Towns' <aj@erisian•com.au>; 'Bitcoin Protocol Discussion'
> <bitcoin-dev@lists•linuxfoundation.org>; 'Gloria Zhao'
> <gloriajzhao@gmail•com>
> Subject: RE: [bitcoin-dev] Package Relay Proposal
> 
> > From: bitcoin-dev <bitcoin-dev-bounces@lists•linuxfoundation.org> On
> Behalf
> > Of Anthony Towns via bitcoin-dev
> > Sent: Wednesday, May 25, 2022 11:56 AM
> 
> > So the other thing is what happens if the peer announcing packages to us
> is
> > dishonest?
> >
> > They announce pkg X, say X has parents A B C and the fee rate is
garbage.
> But
> > actually X has parent D and the fee rate is excellent. Do we request the
> > package from another peer, or every peer, to double check? Otherwise
> we're
> > allowing the first peer we ask about a package to censor that tx from
us?
> >
> > I think the fix for that is just to provide the fee and weight when
> announcing
> > the package rather than only being asked for its info? Then if one peer
> makes
> > it sound like a good deal you ask for the parent txids from them,
dedupe,
> > request, and verify they were honest about the parents.
> 
> Single tx broadcasts do not carry an advertised fee rate, however the'
> feefilter' message (BIP133) provides this distinction. This should be
> interpreted as applicable to packages. Given this message there is no
reason
> to send a (potentially bogus) fee rate with every package. It can only be
> validated by obtaining the full set of txs, and the only recourse is
> dropping (etc.) the peer, as is the case with single txs. Relying on the
> existing message is simpler, more consistent, and more efficient.
> 
> > >> Is it plausible to add the graph in?
> >
> > Likewise, I think you'd have to have the graph info from many nodes if
> you're
> > going to make decisions based on it and don't want hostile peers to be
> able to
> > trick you into ignoring txs.
> >
> > Other idea: what if you encode the parent txs as a short hash of the
wtxid
> > (something like bip152 short ids? perhaps seeded per peer so collisions
> will
> > be different per peer?) and include that in the inv announcement? Would
> > that work to avoid a round trip almost all of the time, while still
giving
> you
> > enough info to save bw by deduping parents?
> 
> As I suggested earlier, a package is fundamentally a compact block (or
> block) announcement without the header. Compact block (BIP152)
> announcement
> is already well-defined and widely implemented. A node should never be
> required to retain an orphan, and BIP152 ensures this is not required.
> 
> Once a validated set of txs within the package has been obtained with
> sufficient fee, a fee-optimal node would accept the largest subgraph of
the
> package that conforms to fee constraints and drop any peer that provides a
> package for which the full graph does not.
> 
> Let us not reinvent the wheel and/or introduce accidental complexity. I
see
> no reason why packaging is not simply BIP152 without the 'header' field,
an
> updated protocol version, and the following sort of changes to names:
> 
> sendpkg
> MSG_CMPCT_PKG
> cmpctpkg
> getpkgtxn
> pkgtxn
> 
> > > For a maximum 25 transactions,
> > >23*24/2 = 276, seems like 36 bytes for a child-with-parents package.
> >
> > If you're doing short ids that's maybe 25*4B=100B already, then the
above
> is
> > up to 36% overhead, I guess. Might be worth thinking more about, but
> maybe
> > more interesting with ancestors than just parents.
> >
> > >Also side note, since there are no size/count params,
> 
> Size is restricted in the same manner as block and transaction broadcasts,
> by consensus. If the fee rate is sufficient there would be no reason to
> preclude any valid size up to what can be mined in one block (packaging
> across blocks is not economically rational under the assumption that one
> miner cannot expect to mine multiple blocks in a row). Count is
incorporated
> into BIP152 as 'shortids_length'.
> 
> > > wondering if we
> > >should just have "version" in "sendpackages" be a bit field instead of
> > >sending a message for each version. 32 versions should be enough right?
> 
> Adding versioning to individual protocols is just a reflection of the
> insufficiency of the initial protocol versioning design, and that of the
> various ad-hoc changes to it (including yet another approach in this
> proposal) that have been introduced to compensate for it, though I'll
> address this in an independent post at some point.
> 
> Best,
> e
> 
> > Maybe but a couple of messages per connection doesn't really seem worth
> > arguing about?
> >
> > Cheers,
> > aj
> >
> >
> > --
> > Sent from my phone.
> > _______________________________________________
> > bitcoin-dev mailing list
> > bitcoin-dev@lists•linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev




 permalink raw reply	[|nested]
2022-05-25 18:55             ` Anthony Towns
  2022-05-25 20:52               ` eric

  
From: Gloria Zhao @ 2022-05-28  1:54 UTC (permalink / raw)
  To: Anthony Towns;  Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 6600 bytes --]

Hi aj, answering slightly out of order:

> what happens if the peer announcing packages to us is dishonest?
> They announce pkg X, say X has parents A B C and the fee rate is garbage.
But actually X has parent D and the fee rate is excellent. Do we request
the package from another peer, or every peer, to double check? Otherwise
we're allowing the first peer we ask about a package to censor that tx from
us?

Yes, providing false information shouldn't be worse than not announcing the
package at all, otherwise we have a censorship vector. In general, the
request logic should not let one peer prevent us from requesting a similar
announcement from another peer.
Yes I was indeed expecting that we would ask for package info from everyone
who announces it until it accepts the package or has full information.
I can see that it's a fair bit of messages (request pckginfo, oh it's low
fee, request pckginfo from somebody else), but we also need to track
announcements / potentially go through the same circle to handle
"notfound"s, right?
In normal running, the fee filter should stop a bunch of honest nodes from
telling us packages that are low fee.

> I think the fix for that is just to provide the fee and weight when
announcing the package rather than only being asked for its info? Then if
one peer makes it sound like a good deal you ask for the parent txids from
them, dedupe, request, and verify they were honest about the parents.
> Likewise, I think you'd have to have the graph info from many nodes if
you're going to make decisions based on it and don't want hostile peers to
be able to trick you into ignoring txs.

I don't think providing more information up front can ever sufficiently
resolve the censorship issue. If we want to prevent any one peer from being
able to censor requests to other peers, we need to store all announcements
and be prepared to request from everybody.

Would it be better if we just took out the fee information and had
"pckginfo" only consist of transaction ids? Sender tries its best to apply
the fee filter? Presumably you have a txInventoryKnown of your peer based
on what they've announced to you... just take the ancestor set of a
transaction, subtract what they already have, and apply the fee filter to
that? Or some kind of algorithm that ensures we don't underestimate? If
it's imperfect, the worst case is the receiver downloads a few transactions
and rejects them. Given that our goal is just to avoid this case, perhaps
opting for simplicity is better than adding a topology graph
serialization/deserialization + feerate assessment algorithm on top of this
protocol...?

>>We'd erroneously ask for A+B+C+X, but really we should only take A+B.
>>But wouldn't A+B also be a package that was announced for B?

> In theory, yes, but maybe it was announced earlier (while our node was
down?) or had dropped from our mempool or similar, either way we don't have
those txs yet.

Hm. It's fine if they have Erlay, since a sender would know in advance that
B is missing and announce it as a package. A potential tack-on solution
would be to request package information whenever you have a "low fee" error
on a parent and "missing inputs" on a child. Or we solve it at the
validation level - instead of submitting each tx individually, we submit
each ancestor subset. Do you think any of these is sufficient? At least the
package properly propagates across nodes which are online when it is
broadcasted...

Best,
Gloria

On Wed, May 25, 2022 at 11:55 AM Anthony Towns <aj@erisian•com.au> wrote:

> On 24 May 2022 5:05:35 pm GMT-04:00, Gloria Zhao via bitcoin-dev <
> bitcoin-dev@lists•linuxfoundation.org> wrote:
> >To clarify, in this situation, I'm imagining something like
> >A: 0 sat, 100vB
> >B: 1500 sat, 100vB
> >C: 0 sat, 100vB
> >X: 500 sat, 100vB
> >feerate floor is 3sat/vB
> >
> >With the algo:
> >>  * is X alone above my fee rate? no, then forget it
> >>  * otherwise, s := X.size, f := X.fees, R := [X]
> >>  * for P = P1..Pn:
> >>   * do I already have P? then skip to the next parent
> >>   * s += P.size, f += P.fees, R += [P]
> >>  * if f/s above my fee rate floor? if so, request all the txs in R
> >
> >We'd erroneously ask for A+B+C+X, but really we should only take A+B.
> >But wouldn't A+B also be a package that was announced for B?
>
> In theory, yes, but maybe it was announced earlier (while our node was
> down?) or had dropped from our mempool or similar, either way we don't have
> those txs yet.
>
> >Please lmk if you were imagining something different. I think I may be
> >missing something.
>
> That's what I was thinking, yes.
>
> So the other thing is what happens if the peer announcing packages to us
> is dishonest?
>
> They announce pkg X, say X has parents A B C and the fee rate is garbage.
> But actually X has parent D and the fee rate is excellent. Do we request
> the package from another peer, or every peer, to double check? Otherwise
> we're allowing the first peer we ask about a package to censor that tx from
> us?
>
> I think the fix for that is just to provide the fee and weight when
> announcing the package rather than only being asked for its info? Then if
> one peer makes it sound like a good deal you ask for the parent txids from
> them, dedupe, request, and verify they were honest about the parents.
>
> >> Is it plausible to add the graph in?
>
> Likewise, I think you'd have to have the graph info from many nodes if
> you're going to make decisions based on it and don't want hostile peers to
> be able to trick you into ignoring txs.
>
> Other idea: what if you encode the parent txs as a short hash of the wtxid
> (something like bip152 short ids? perhaps seeded per peer so collisions
> will be different per peer?) and include that in the inv announcement?
> Would that work to avoid a round trip almost all of the time, while still
> giving you enough info to save bw by deduping parents?
>
>
> > For a maximum 25 transactions,
> >23*24/2 = 276, seems like 36 bytes for a child-with-parents package.
>
> If you're doing short ids that's maybe 25*4B=100B already, then the above
> is up to 36% overhead, I guess. Might be worth thinking more about, but
> maybe more interesting with ancestors than just parents.
>
> >Also side note, since there are no size/count params, wondering if we
> >should just have "version" in "sendpackages" be a bit field instead of
> >sending a message for each version. 32 versions should be enough right?
>
> Maybe but a couple of messages per connection doesn't really seem worth
> arguing about?
>
> Cheers,
> aj
>
>
> --
> Sent from my phone.
>

[-- Attachment #2: Type: text/html, Size: 8618 bytes --]

 permalink raw reply	[|nested]
2022-05-26  2:59                 ` eric

  2022-06-08 15:59                     ` Suhas Daftuar
  
From: Gloria Zhao @ 2022-06-07 17:44 UTC (permalink / raw)
  To: eric;  Bitcoin Protocol Discussion, Anthony Towns

[-- Attachment #1: Type: text/plain, Size: 10022 bytes --]

Hi Eric, aj, all,

Sorry for the delayed response. @aj I'm including some paraphrased points
from our offline discussion (thanks).

> Other idea: what if you encode the parent txs as a short hash of the
wtxid (something like bip152 short ids? perhaps seeded per peer so
collisions will be different per peer?) and include that in the inv
announcement? Would that work to avoid a round trip almost all of the time,
while still giving you enough info to save bw by deduping parents?

> As I suggested earlier, a package is fundamentally a compact block (or
> block) announcement without the header. Compact block (BIP152)
announcement
> is already well-defined and widely implemented...

> Let us not reinvent the wheel and/or introduce accidental complexity. I
see
> no reason why packaging is not simply BIP152 without the 'header' field,
an
> updated protocol version, and the following sort of changes to names

Interestingly, "why not use BIP 152 shortids to save bandwidth?" is by far
the most common suggestion I hear (including offline feedback). Here's a
full explanation:

BIP 152 shortens transaction hashes (32 bytes) to shortids (6 bytes) to
save a significant amount of network bandwidth, which is extremely
important in block relay. However, this comes at the expense of
computational complexity. There is no way to directly calculate a
transaction hash from a shortid; upon receipt of a compact block, a node is
expected to calculate the shortids of every unconfirmed transaction it
knows about to find the matches (BIP 152: [1], Bitcoin Core: [2]). This is
expensive but appropriate for block relay, since the block must have a
valid Proof of Work and new blocks only come every ~10 minutes. On the
other hand, if we require nodes to calculate shortids for every transaction
in their mempools every time they receive a package, we are creating a DoS
vector. Unconfirmed transactions don't need PoW and, to have a live
transaction relay network, we should expect nodes to handle transactions at
a high-ish rate (i.e. at least 1000s of times more transactions than
blocks). We can't pre-calculate or cache shortids for mempool transactions,
since the SipHash key depends on the block hash and a per-connection salt.

Additionally, shortid calculation is not designed to prevent intentional
individual collisions. If we were to use these shortids to deduplicate
transactions we've supposedly already seen, we may have a censorship
vector. Again, these tradeoffs make sense for compact block relay (see
shortid section in BIP 152 [3]), but not package relay.

TLDR: DoSy if we calculate shortids on every package and censorship vector
if we use shortids for deduplication.

> Given this message there is no reason
> to send a (potentially bogus) fee rate with every package. It can only be

> validated by obtaining the full set of txs, and the only recourse is
> dropping (etc.) the peer, as is the case with single txs.

Yeah, I agree with this. Combined with the previous discussion with aj
(i.e. we can't accurately communicate the incentive compatibility of a
package without sending the full graph, and this whole dance is to avoid
downloading a few low-fee transactions in uncommon edge cases), I've
realized I should remove the fee + weight information from pkginfo. Yay for
less complexity!

Also, this might be pedantic, but I said something incorrect earlier and
would like to correct myself:

>> In theory, yes, but maybe it was announced earlier (while our node was
down?) or had dropped from our mempool or similar, either way we don't have
those txs yet.

I said "It's fine if they have Erlay, since a sender would know in advance
that B is missing and announce it as a package." But this isn't true since
we're only using reconciliation in place of flooding to announce
transactions as they arrive, not for rebroadcast, and we're not doing full
mempool set reconciliation. In any case, making sure a node receives the
transactions announced when it was offline is not something we guarantee,
not an intended use case for package relay, and not worsened by this.

Thanks for your feedback!

Best,
Gloria

[1]:
https://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki#cmpctblock
[2]:
https://github.com/bitcoin/bitcoin/blob/master/src/blockencodings.cpp#L49
[3]:
https://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki#short-transaction-id-calculation

On Thu, May 26, 2022 at 3:59 AM <eric@voskuil•org> wrote:

> Given that packages have no header, the package requires identity in a
> BIP152 scheme. For example 'header' and 'blockhash' fields can be replaced
> with a Merkle root (e.g. "identity" field) for the package, uniquely
> identifying the partially-ordered set of txs. And use of 'getdata' (to
> obtain a package by hash) can be eliminated (not a use case).
>
> e
>
> > -----Original Message-----
> > From: eric@voskuil•org <eric@voskuil•org>
> > Sent: Wednesday, May 25, 2022 1:52 PM
> > To: 'Anthony Towns' <aj@erisian•com.au>; 'Bitcoin Protocol Discussion'
> > <bitcoin-dev@lists•linuxfoundation.org>; 'Gloria Zhao'
> > <gloriajzhao@gmail•com>
> > Subject: RE: [bitcoin-dev] Package Relay Proposal
> >
> > > From: bitcoin-dev <bitcoin-dev-bounces@lists•linuxfoundation.org> On
> > Behalf
> > > Of Anthony Towns via bitcoin-dev
> > > Sent: Wednesday, May 25, 2022 11:56 AM
> >
> > > So the other thing is what happens if the peer announcing packages to
> us
> > is
> > > dishonest?
> > >
> > > They announce pkg X, say X has parents A B C and the fee rate is
> garbage.
> > But
> > > actually X has parent D and the fee rate is excellent. Do we request
> the
> > > package from another peer, or every peer, to double check? Otherwise
> > we're
> > > allowing the first peer we ask about a package to censor that tx from
> us?
> > >
> > > I think the fix for that is just to provide the fee and weight when
> > announcing
> > > the package rather than only being asked for its info? Then if one peer
> > makes
> > > it sound like a good deal you ask for the parent txids from them,
> dedupe,
> > > request, and verify they were honest about the parents.
> >
> > Single tx broadcasts do not carry an advertised fee rate, however the'
> > feefilter' message (BIP133) provides this distinction. This should be
> > interpreted as applicable to packages. Given this message there is no
> reason
> > to send a (potentially bogus) fee rate with every package. It can only be
> > validated by obtaining the full set of txs, and the only recourse is
> > dropping (etc.) the peer, as is the case with single txs. Relying on the
> > existing message is simpler, more consistent, and more efficient.
> >
> > > >> Is it plausible to add the graph in?
> > >
> > > Likewise, I think you'd have to have the graph info from many nodes if
> > you're
> > > going to make decisions based on it and don't want hostile peers to be
> > able to
> > > trick you into ignoring txs.
> > >
> > > Other idea: what if you encode the parent txs as a short hash of the
> wtxid
> > > (something like bip152 short ids? perhaps seeded per peer so collisions
> > will
> > > be different per peer?) and include that in the inv announcement? Would
> > > that work to avoid a round trip almost all of the time, while still
> giving
> > you
> > > enough info to save bw by deduping parents?
> >
> > As I suggested earlier, a package is fundamentally a compact block (or
> > block) announcement without the header. Compact block (BIP152)
> > announcement
> > is already well-defined and widely implemented. A node should never be
> > required to retain an orphan, and BIP152 ensures this is not required.
> >
> > Once a validated set of txs within the package has been obtained with
> > sufficient fee, a fee-optimal node would accept the largest subgraph of
> the
> > package that conforms to fee constraints and drop any peer that provides
> a
> > package for which the full graph does not.
> >
> > Let us not reinvent the wheel and/or introduce accidental complexity. I
> see
> > no reason why packaging is not simply BIP152 without the 'header' field,
> an
> > updated protocol version, and the following sort of changes to names:
> >
> > sendpkg
> > MSG_CMPCT_PKG
> > cmpctpkg
> > getpkgtxn
> > pkgtxn
> >
> > > > For a maximum 25 transactions,
> > > >23*24/2 = 276, seems like 36 bytes for a child-with-parents package.
> > >
> > > If you're doing short ids that's maybe 25*4B=100B already, then the
> above
> > is
> > > up to 36% overhead, I guess. Might be worth thinking more about, but
> > maybe
> > > more interesting with ancestors than just parents.
> > >
> > > >Also side note, since there are no size/count params,
> >
> > Size is restricted in the same manner as block and transaction
> broadcasts,
> > by consensus. If the fee rate is sufficient there would be no reason to
> > preclude any valid size up to what can be mined in one block (packaging
> > across blocks is not economically rational under the assumption that one
> > miner cannot expect to mine multiple blocks in a row). Count is
> incorporated
> > into BIP152 as 'shortids_length'.
> >
> > > > wondering if we
> > > >should just have "version" in "sendpackages" be a bit field instead of
> > > >sending a message for each version. 32 versions should be enough
> right?
> >
> > Adding versioning to individual protocols is just a reflection of the
> > insufficiency of the initial protocol versioning design, and that of the
> > various ad-hoc changes to it (including yet another approach in this
> > proposal) that have been introduced to compensate for it, though I'll
> > address this in an independent post at some point.
> >
> > Best,
> > e
> >
> > > Maybe but a couple of messages per connection doesn't really seem worth
> > > arguing about?
> > >
> > > Cheers,
> > > aj
> > >
> > >
> > > --
> > > Sent from my phone.
> > > _______________________________________________
> > > bitcoin-dev mailing list
> > > bitcoin-dev@lists•linuxfoundation.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>
>

[-- Attachment #2: Type: text/html, Size: 12920 bytes --]

 permalink raw reply	[|nested]
2022-06-07 17:44                   ` Gloria Zhao

  2022-06-14  9:59                       ` Gloria Zhao
  
From: Suhas Daftuar @ 2022-06-08 15:59 UTC (permalink / raw)
  To: Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 15911 bytes --]

Hi,

Thanks again for your work on this!

One question I have is about potential bandwidth waste in the case of nodes
running with different policy rules.  Here's my understanding of a scenario
I think could happen:

1) Transaction A is both low-fee and non-standard to some nodes on the
network.
2) Whenever a transaction T that spends A is relayed, new nodes will send
INV(PKGINFO1, T) to all package-relay peers.
3) Nodes on the network that have implemented package relay, but do not
accept A, will send getdata(PKGINFO1, T) and learn all of T's unconfirmed
parents (~32 bytes * number of parents(T)).
4) Such nodes will reject T.  But because of transaction malleability, and
to avoid being blinded to a transaction unnecessarily, these nodes will
likely still send getdata(PKGINFO1, T) to every node that announces T, in
case someone has a transaction that includes an alternate set of parent
transactions that would pass policy checks.

Is that understanding correct?  I think a good design goal would be to not
waste bandwidth in non-adversarial situations.  In this case, there would
be bandwidth waste from downloading duplicate data from all your peers,
just because the announcement doesn't commit to the set of parent wtxids
that we'd get from the peer (and so we are unable to determine that all our
peers would be telling us the same thing, just based on the announcement).

Some ways to mitigate this might be to: (a) include a hash (maybe even just
a 20-byte hash -- is that enough security?) of the package wtxids (in some
canonical ordering) along with the wtxid of the child in the initial
announcement; (b) limit the use of v1 packages to transactions with very
few parents (I don't know if this is reasonable for the use cases we have
in mind).

Another point I wanted to bring up is about the rules around v1 package
validation generally, and the use of a blockhash in transaction relay
specifically.  My first observation is that it won't always be the case
that a v1 package relay node will be able to validate that a set of package
transactions is fully sorted topologically, because there may be
(non-parent) ancestors that are missing from the package and the best a
peer can validate is topology within the package -- this means that a peer
can validly (under this BIP) relay transaction packages out of the true
topological sort (if all ancestors were included).

This makes me wonder how useful this topological rule is.  I suppose there
is some value in preventing completely broken implementations from staying
connected and so there is no harm in having the rule, but perhaps it would
be helpful to add that nodes SHOULD order transactions based on topological
sort in the complete transaction graph, so that if missing-from-package
ancestors are already known by a peer (which is the expected case when
using v1 package relay on transactions that have more than one generation
of unconfirmed ancestor) then the remaining transactions are already
properly ordered, and this is helpful even if unenforceable in general.

The other observation I wanted to make was that having transaction relay
gated on whether two nodes agree on chain tip seems like an overly
restrictive criteria.  I think an important design principle is that we
want to minimize disruption from network splits -- if there are competing
blocks found in a small window of time, it's likely that the utxo set is
not materially different on the two chains (assuming miners are selecting
from roughly the same sets of transactions when this happens, which is
typical).  Having transaction relay bifurcate on the two network halves
would seem to exacerbate the difference between the two sides of the split
-- users ought to be agnostic about how benign splits are resolved and
would likely want their transactions to relay across the whole network.

Additionally, use of a chain tip might impose a larger burden than is
necessary on software that would seek to participate in transaction relay
without implementing headers sync/validation.  I don't know what software
exists on the network, but I imagine there are a lot of scripts out there
for transaction submission to the public p2p network, and in thinking
about modifying such a script to utilize package relay it seems like an
unnecessary added burden to first learn a node's tip before trying to relay
a transaction.

Could you explain again what the benefit of including the blockhash is?  It
seems like it is just so that a node could prioritize transaction relay
from peers with the same chain tip to maximize the likelihood of
transaction acceptance, but in the common case this seems like a pretty
negligible concern, and in the case of a chain fork that persists for many
minutes it seems better to me that we not partition the network into
package-relay regimes and just risk a little extra bandwidth in one
direction or the other.  If we solve the problem I brought up at the
beginning (of de-duplicating package data across peers with a
package-wtxid-commitment in the announcement), I think this is just some
wasted pkginfo bandwidth on a single-link, and not across links (as we
could cache validation failure for a package-hash to avoid re-requesting
duplicate pkginfo1 messages).

Best,
Suhas


On Tue, Jun 7, 2022 at 1:57 PM Gloria Zhao via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> wrote:

> Hi Eric, aj, all,
>
> Sorry for the delayed response. @aj I'm including some paraphrased points
> from our offline discussion (thanks).
>
> > Other idea: what if you encode the parent txs as a short hash of the
> wtxid (something like bip152 short ids? perhaps seeded per peer so
> collisions will be different per peer?) and include that in the inv
> announcement? Would that work to avoid a round trip almost all of the time,
> while still giving you enough info to save bw by deduping parents?
>
> > As I suggested earlier, a package is fundamentally a compact block (or
> > block) announcement without the header. Compact block (BIP152)
> announcement
> > is already well-defined and widely implemented...
>
> > Let us not reinvent the wheel and/or introduce accidental complexity. I
> see
> > no reason why packaging is not simply BIP152 without the 'header' field,
> an
> > updated protocol version, and the following sort of changes to names
>
> Interestingly, "why not use BIP 152 shortids to save bandwidth?" is by far
> the most common suggestion I hear (including offline feedback). Here's a
> full explanation:
>
> BIP 152 shortens transaction hashes (32 bytes) to shortids (6 bytes) to
> save a significant amount of network bandwidth, which is extremely
> important in block relay. However, this comes at the expense of
> computational complexity. There is no way to directly calculate a
> transaction hash from a shortid; upon receipt of a compact block, a node is
> expected to calculate the shortids of every unconfirmed transaction it
> knows about to find the matches (BIP 152: [1], Bitcoin Core: [2]). This is
> expensive but appropriate for block relay, since the block must have a
> valid Proof of Work and new blocks only come every ~10 minutes. On the
> other hand, if we require nodes to calculate shortids for every transaction
> in their mempools every time they receive a package, we are creating a DoS
> vector. Unconfirmed transactions don't need PoW and, to have a live
> transaction relay network, we should expect nodes to handle transactions at
> a high-ish rate (i.e. at least 1000s of times more transactions than
> blocks). We can't pre-calculate or cache shortids for mempool transactions,
> since the SipHash key depends on the block hash and a per-connection salt.
>
> Additionally, shortid calculation is not designed to prevent intentional
> individual collisions. If we were to use these shortids to deduplicate
> transactions we've supposedly already seen, we may have a censorship
> vector. Again, these tradeoffs make sense for compact block relay (see
> shortid section in BIP 152 [3]), but not package relay.
>
> TLDR: DoSy if we calculate shortids on every package and censorship vector
> if we use shortids for deduplication.
>
> > Given this message there is no reason
> > to send a (potentially bogus) fee rate with every package. It can only
> be
> > validated by obtaining the full set of txs, and the only recourse is
> > dropping (etc.) the peer, as is the case with single txs.
>
> Yeah, I agree with this. Combined with the previous discussion with aj
> (i.e. we can't accurately communicate the incentive compatibility of a
> package without sending the full graph, and this whole dance is to avoid
> downloading a few low-fee transactions in uncommon edge cases), I've
> realized I should remove the fee + weight information from pkginfo. Yay for
> less complexity!
>
> Also, this might be pedantic, but I said something incorrect earlier and
> would like to correct myself:
>
> >> In theory, yes, but maybe it was announced earlier (while our node was
> down?) or had dropped from our mempool or similar, either way we don't have
> those txs yet.
>
> I said "It's fine if they have Erlay, since a sender would know in advance
> that B is missing and announce it as a package." But this isn't true since
> we're only using reconciliation in place of flooding to announce
> transactions as they arrive, not for rebroadcast, and we're not doing full
> mempool set reconciliation. In any case, making sure a node receives the
> transactions announced when it was offline is not something we guarantee,
> not an intended use case for package relay, and not worsened by this.
>
> Thanks for your feedback!
>
> Best,
> Gloria
>
> [1]:
> https://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki#cmpctblock
> [2]:
> https://github.com/bitcoin/bitcoin/blob/master/src/blockencodings.cpp#L49
> [3]:
> https://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki#short-transaction-id-calculation
>
> On Thu, May 26, 2022 at 3:59 AM <eric@voskuil•org> wrote:
>
>> Given that packages have no header, the package requires identity in a
>> BIP152 scheme. For example 'header' and 'blockhash' fields can be replaced
>> with a Merkle root (e.g. "identity" field) for the package, uniquely
>> identifying the partially-ordered set of txs. And use of 'getdata' (to
>> obtain a package by hash) can be eliminated (not a use case).
>>
>> e
>>
>> > -----Original Message-----
>> > From: eric@voskuil•org <eric@voskuil•org>
>> > Sent: Wednesday, May 25, 2022 1:52 PM
>> > To: 'Anthony Towns' <aj@erisian•com.au>; 'Bitcoin Protocol Discussion'
>> > <bitcoin-dev@lists•linuxfoundation.org>; 'Gloria Zhao'
>> > <gloriajzhao@gmail•com>
>> > Subject: RE: [bitcoin-dev] Package Relay Proposal
>> >
>> > > From: bitcoin-dev <bitcoin-dev-bounces@lists•linuxfoundation.org> On
>> > Behalf
>> > > Of Anthony Towns via bitcoin-dev
>> > > Sent: Wednesday, May 25, 2022 11:56 AM
>> >
>> > > So the other thing is what happens if the peer announcing packages to
>> us
>> > is
>> > > dishonest?
>> > >
>> > > They announce pkg X, say X has parents A B C and the fee rate is
>> garbage.
>> > But
>> > > actually X has parent D and the fee rate is excellent. Do we request
>> the
>> > > package from another peer, or every peer, to double check? Otherwise
>> > we're
>> > > allowing the first peer we ask about a package to censor that tx from
>> us?
>> > >
>> > > I think the fix for that is just to provide the fee and weight when
>> > announcing
>> > > the package rather than only being asked for its info? Then if one
>> peer
>> > makes
>> > > it sound like a good deal you ask for the parent txids from them,
>> dedupe,
>> > > request, and verify they were honest about the parents.
>> >
>> > Single tx broadcasts do not carry an advertised fee rate, however the'
>> > feefilter' message (BIP133) provides this distinction. This should be
>> > interpreted as applicable to packages. Given this message there is no
>> reason
>> > to send a (potentially bogus) fee rate with every package. It can only
>> be
>> > validated by obtaining the full set of txs, and the only recourse is
>> > dropping (etc.) the peer, as is the case with single txs. Relying on the
>> > existing message is simpler, more consistent, and more efficient.
>> >
>> > > >> Is it plausible to add the graph in?
>> > >
>> > > Likewise, I think you'd have to have the graph info from many nodes if
>> > you're
>> > > going to make decisions based on it and don't want hostile peers to be
>> > able to
>> > > trick you into ignoring txs.
>> > >
>> > > Other idea: what if you encode the parent txs as a short hash of the
>> wtxid
>> > > (something like bip152 short ids? perhaps seeded per peer so
>> collisions
>> > will
>> > > be different per peer?) and include that in the inv announcement?
>> Would
>> > > that work to avoid a round trip almost all of the time, while still
>> giving
>> > you
>> > > enough info to save bw by deduping parents?
>> >
>> > As I suggested earlier, a package is fundamentally a compact block (or
>> > block) announcement without the header. Compact block (BIP152)
>> > announcement
>> > is already well-defined and widely implemented. A node should never be
>> > required to retain an orphan, and BIP152 ensures this is not required.
>> >
>> > Once a validated set of txs within the package has been obtained with
>> > sufficient fee, a fee-optimal node would accept the largest subgraph of
>> the
>> > package that conforms to fee constraints and drop any peer that
>> provides a
>> > package for which the full graph does not.
>> >
>> > Let us not reinvent the wheel and/or introduce accidental complexity. I
>> see
>> > no reason why packaging is not simply BIP152 without the 'header' field,
>> an
>> > updated protocol version, and the following sort of changes to names:
>> >
>> > sendpkg
>> > MSG_CMPCT_PKG
>> > cmpctpkg
>> > getpkgtxn
>> > pkgtxn
>> >
>> > > > For a maximum 25 transactions,
>> > > >23*24/2 = 276, seems like 36 bytes for a child-with-parents package.
>> > >
>> > > If you're doing short ids that's maybe 25*4B=100B already, then the
>> above
>> > is
>> > > up to 36% overhead, I guess. Might be worth thinking more about, but
>> > maybe
>> > > more interesting with ancestors than just parents.
>> > >
>> > > >Also side note, since there are no size/count params,
>> >
>> > Size is restricted in the same manner as block and transaction
>> broadcasts,
>> > by consensus. If the fee rate is sufficient there would be no reason to
>> > preclude any valid size up to what can be mined in one block (packaging
>> > across blocks is not economically rational under the assumption that one
>> > miner cannot expect to mine multiple blocks in a row). Count is
>> incorporated
>> > into BIP152 as 'shortids_length'.
>> >
>> > > > wondering if we
>> > > >should just have "version" in "sendpackages" be a bit field instead
>> of
>> > > >sending a message for each version. 32 versions should be enough
>> right?
>> >
>> > Adding versioning to individual protocols is just a reflection of the
>> > insufficiency of the initial protocol versioning design, and that of the
>> > various ad-hoc changes to it (including yet another approach in this
>> > proposal) that have been introduced to compensate for it, though I'll
>> > address this in an independent post at some point.
>> >
>> > Best,
>> > e
>> >
>> > > Maybe but a couple of messages per connection doesn't really seem
>> worth
>> > > arguing about?
>> > >
>> > > Cheers,
>> > > aj
>> > >
>> > >
>> > > --
>> > > Sent from my phone.
>> > > _______________________________________________
>> > > bitcoin-dev mailing list
>> > > bitcoin-dev@lists•linuxfoundation.org
>> > > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
>> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 19351 bytes --]

 permalink raw reply	[|nested]
2022-06-08 15:59                     ` Suhas Daftuar

  
From: Gloria Zhao @ 2022-06-14  9:59 UTC (permalink / raw)
  To: Suhas Daftuar, Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 20866 bytes --]

Hi Suhas,

Thanks for your attention and feedback!

> Transaction A is both low-fee and non-standard to some nodes on the
network...
> ...Whenever a transaction T that spends A is relayed, new nodes will send
INV(PKGINFO1, T) to all package-relay peers...
> ...because of transaction malleability, and to avoid being blinded to a
transaction unnecessarily, these nodes will likely still send
getdata(PKGINFO1, T) to every node that announces T...

Yes, we'd request pkginfo unless we already had the transaction in our
mempool. The pkginfo step is intended to prevent nodes from ever
downloading a transaction more than once; I was going for a benchmark of
"packages are announced once per p2p connection, transaction data
downloaded once per node".

In this scenario, both A and T's wtxids would be sent once per p2p
connection and transaction data downloaded once per node. If T has other
unconfirmed parents, the low-fee ones will only be announced once (in
pkginfo) per link. If it has high-fee parents, they will indeed be
announced more than once per link (once individually, then again in
pkginfo).

More precisely: if a package contains any transactions which are
non-standard to one peer and standard to another, the package transactions
(parents, not child) that pass the fee filter on their own will be
announced twice instead of once.

> I think a good design goal would be to not waste bandwidth in
non-adversarial situations. In this case, there would be bandwidth waste
from downloading duplicate data from all your peers, just because the
announcement doesn't commit to the set of parent wtxids that we'd get from
the peer (and so we are unable to determine that all our peers would be
telling us the same thing, just based on the announcement).

Each transaction is only downloaded once per node here, and each package
announced/pkginfo sent once per link. I definitely understand that this
doesn't pass a benchmark of "every transaction is announced at most once
per link," but it's still on the magnitude of 32-byte hashes. Adding a
commitment to parents in the announcements is an extra hash per link in all
cases - my question is whether it's worth it? We'd also need to write new
inv/getdata message types for package relay, though that's probably a
weaker argument.

> it won't always be the case that a v1 package relay node will be able to
validate that a set of package transactions is fully sorted topologically,
because there may be (non-parent) ancestors that are missing from the
package and the best a peer can validate is topology within the package --
this means that a peer can validly (under this BIP) relay transaction
packages out of the true topological sort (if all ancestors were included).

Good point. Since v1 packages don't necessarily include the full ancestor
set, we wouldn't be able to verify that two parents are in the right order
if they have an indirect dependency, e.g. parent 1 spends a tx
("grandparent") which spends parent 2. Note that the grandparent couldn't
possibly be in the mempool unless parent 2 is. We'd eventually get
everything submitted as long as we received the grandparent, and then know
whether the package was topologically sorted. But I think you're right that
this could be a "nice to have" instead of a protocol requirement.

> Could you explain again what the benefit of including the blockhash is?
It seems like it is just so that a node could prioritize transaction relay
from peers with the same chain tip to maximize the likelihood of
transaction acceptance, but in the common case this seems like a pretty
negligible concern...

The blockhash is necessary in order to disambiguate between a malformed
package and difference in chain tip. If a parent is missing from a package,
it's possible it was mined in a recent block that we haven't seen yet.
Validating using a UTXO set, all we see is "missing inputs" when we try to
validate the child; we wouldn't know if our peer had sent us a malformed
package or if we were behind.

I'm hoping some of these clarifications are helpful to post publicly, but I
know I haven't fully addressed all the concerns you've brought up. Will
continue to think about this.

Best,
Gloria

On Wed, Jun 8, 2022 at 4:59 PM Suhas Daftuar via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> wrote:

> Hi,
>
> Thanks again for your work on this!
>
> One question I have is about potential bandwidth waste in the case of
> nodes running with different policy rules.  Here's my understanding of a
> scenario I think could happen:
>
> 1) Transaction A is both low-fee and non-standard to some nodes on the
> network.
> 2) Whenever a transaction T that spends A is relayed, new nodes will send
> INV(PKGINFO1, T) to all package-relay peers.
> 3) Nodes on the network that have implemented package relay, but do not
> accept A, will send getdata(PKGINFO1, T) and learn all of T's unconfirmed
> parents (~32 bytes * number of parents(T)).
> 4) Such nodes will reject T.  But because of transaction malleability, and
> to avoid being blinded to a transaction unnecessarily, these nodes will
> likely still send getdata(PKGINFO1, T) to every node that announces T, in
> case someone has a transaction that includes an alternate set of parent
> transactions that would pass policy checks.
>
> Is that understanding correct?  I think a good design goal would be to not
> waste bandwidth in non-adversarial situations.  In this case, there would
> be bandwidth waste from downloading duplicate data from all your peers,
> just because the announcement doesn't commit to the set of parent wtxids
> that we'd get from the peer (and so we are unable to determine that all our
> peers would be telling us the same thing, just based on the announcement).
>
> Some ways to mitigate this might be to: (a) include a hash (maybe even
> just a 20-byte hash -- is that enough security?) of the package wtxids (in
> some canonical ordering) along with the wtxid of the child in the initial
> announcement; (b) limit the use of v1 packages to transactions with very
> few parents (I don't know if this is reasonable for the use cases we have
> in mind).
>
> Another point I wanted to bring up is about the rules around v1 package
> validation generally, and the use of a blockhash in transaction relay
> specifically.  My first observation is that it won't always be the case
> that a v1 package relay node will be able to validate that a set of package
> transactions is fully sorted topologically, because there may be
> (non-parent) ancestors that are missing from the package and the best a
> peer can validate is topology within the package -- this means that a peer
> can validly (under this BIP) relay transaction packages out of the true
> topological sort (if all ancestors were included).
>
> This makes me wonder how useful this topological rule is.  I suppose there
> is some value in preventing completely broken implementations from staying
> connected and so there is no harm in having the rule, but perhaps it would
> be helpful to add that nodes SHOULD order transactions based on topological
> sort in the complete transaction graph, so that if missing-from-package
> ancestors are already known by a peer (which is the expected case when
> using v1 package relay on transactions that have more than one generation
> of unconfirmed ancestor) then the remaining transactions are already
> properly ordered, and this is helpful even if unenforceable in general.
>
> The other observation I wanted to make was that having transaction relay
> gated on whether two nodes agree on chain tip seems like an overly
> restrictive criteria.  I think an important design principle is that we
> want to minimize disruption from network splits -- if there are competing
> blocks found in a small window of time, it's likely that the utxo set is
> not materially different on the two chains (assuming miners are selecting
> from roughly the same sets of transactions when this happens, which is
> typical).  Having transaction relay bifurcate on the two network halves
> would seem to exacerbate the difference between the two sides of the split
> -- users ought to be agnostic about how benign splits are resolved and
> would likely want their transactions to relay across the whole network.
>
> Additionally, use of a chain tip might impose a larger burden than is
> necessary on software that would seek to participate in transaction relay
> without implementing headers sync/validation.  I don't know what software
> exists on the network, but I imagine there are a lot of scripts out there
> for transaction submission to the public p2p network, and in thinking
> about modifying such a script to utilize package relay it seems like an
> unnecessary added burden to first learn a node's tip before trying to relay
> a transaction.
>
> Could you explain again what the benefit of including the blockhash is?
> It seems like it is just so that a node could prioritize transaction relay
> from peers with the same chain tip to maximize the likelihood of
> transaction acceptance, but in the common case this seems like a pretty
> negligible concern, and in the case of a chain fork that persists for many
> minutes it seems better to me that we not partition the network into
> package-relay regimes and just risk a little extra bandwidth in one
> direction or the other.  If we solve the problem I brought up at the
> beginning (of de-duplicating package data across peers with a
> package-wtxid-commitment in the announcement), I think this is just some
> wasted pkginfo bandwidth on a single-link, and not across links (as we
> could cache validation failure for a package-hash to avoid re-requesting
> duplicate pkginfo1 messages).
>
> Best,
> Suhas
>
>
> On Tue, Jun 7, 2022 at 1:57 PM Gloria Zhao via bitcoin-dev <
> bitcoin-dev@lists•linuxfoundation.org> wrote:
>
>> Hi Eric, aj, all,
>>
>> Sorry for the delayed response. @aj I'm including some paraphrased points
>> from our offline discussion (thanks).
>>
>> > Other idea: what if you encode the parent txs as a short hash of the
>> wtxid (something like bip152 short ids? perhaps seeded per peer so
>> collisions will be different per peer?) and include that in the inv
>> announcement? Would that work to avoid a round trip almost all of the time,
>> while still giving you enough info to save bw by deduping parents?
>>
>> > As I suggested earlier, a package is fundamentally a compact block (or
>> > block) announcement without the header. Compact block (BIP152)
>> announcement
>> > is already well-defined and widely implemented...
>>
>> > Let us not reinvent the wheel and/or introduce accidental complexity. I
>> see
>> > no reason why packaging is not simply BIP152 without the 'header'
>> field, an
>> > updated protocol version, and the following sort of changes to names
>>
>> Interestingly, "why not use BIP 152 shortids to save bandwidth?" is by
>> far the most common suggestion I hear (including offline feedback). Here's
>> a full explanation:
>>
>> BIP 152 shortens transaction hashes (32 bytes) to shortids (6 bytes) to
>> save a significant amount of network bandwidth, which is extremely
>> important in block relay. However, this comes at the expense of
>> computational complexity. There is no way to directly calculate a
>> transaction hash from a shortid; upon receipt of a compact block, a node is
>> expected to calculate the shortids of every unconfirmed transaction it
>> knows about to find the matches (BIP 152: [1], Bitcoin Core: [2]). This is
>> expensive but appropriate for block relay, since the block must have a
>> valid Proof of Work and new blocks only come every ~10 minutes. On the
>> other hand, if we require nodes to calculate shortids for every transaction
>> in their mempools every time they receive a package, we are creating a DoS
>> vector. Unconfirmed transactions don't need PoW and, to have a live
>> transaction relay network, we should expect nodes to handle transactions at
>> a high-ish rate (i.e. at least 1000s of times more transactions than
>> blocks). We can't pre-calculate or cache shortids for mempool transactions,
>> since the SipHash key depends on the block hash and a per-connection salt.
>>
>> Additionally, shortid calculation is not designed to prevent intentional
>> individual collisions. If we were to use these shortids to deduplicate
>> transactions we've supposedly already seen, we may have a censorship
>> vector. Again, these tradeoffs make sense for compact block relay (see
>> shortid section in BIP 152 [3]), but not package relay.
>>
>> TLDR: DoSy if we calculate shortids on every package and censorship
>> vector if we use shortids for deduplication.
>>
>> > Given this message there is no reason
>> > to send a (potentially bogus) fee rate with every package. It can only
>> be
>> > validated by obtaining the full set of txs, and the only recourse is
>> > dropping (etc.) the peer, as is the case with single txs.
>>
>> Yeah, I agree with this. Combined with the previous discussion with aj
>> (i.e. we can't accurately communicate the incentive compatibility of a
>> package without sending the full graph, and this whole dance is to avoid
>> downloading a few low-fee transactions in uncommon edge cases), I've
>> realized I should remove the fee + weight information from pkginfo. Yay for
>> less complexity!
>>
>> Also, this might be pedantic, but I said something incorrect earlier and
>> would like to correct myself:
>>
>> >> In theory, yes, but maybe it was announced earlier (while our node was
>> down?) or had dropped from our mempool or similar, either way we don't have
>> those txs yet.
>>
>> I said "It's fine if they have Erlay, since a sender would know in
>> advance that B is missing and announce it as a package." But this isn't
>> true since we're only using reconciliation in place of flooding to announce
>> transactions as they arrive, not for rebroadcast, and we're not doing full
>> mempool set reconciliation. In any case, making sure a node receives the
>> transactions announced when it was offline is not something we guarantee,
>> not an intended use case for package relay, and not worsened by this.
>>
>> Thanks for your feedback!
>>
>> Best,
>> Gloria
>>
>> [1]:
>> https://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki#cmpctblock
>> [2]:
>> https://github.com/bitcoin/bitcoin/blob/master/src/blockencodings.cpp#L49
>> [3]:
>> https://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki#short-transaction-id-calculation
>>
>> On Thu, May 26, 2022 at 3:59 AM <eric@voskuil•org> wrote:
>>
>>> Given that packages have no header, the package requires identity in a
>>> BIP152 scheme. For example 'header' and 'blockhash' fields can be
>>> replaced
>>> with a Merkle root (e.g. "identity" field) for the package, uniquely
>>> identifying the partially-ordered set of txs. And use of 'getdata' (to
>>> obtain a package by hash) can be eliminated (not a use case).
>>>
>>> e
>>>
>>> > -----Original Message-----
>>> > From: eric@voskuil•org <eric@voskuil•org>
>>> > Sent: Wednesday, May 25, 2022 1:52 PM
>>> > To: 'Anthony Towns' <aj@erisian•com.au>; 'Bitcoin Protocol Discussion'
>>> > <bitcoin-dev@lists•linuxfoundation.org>; 'Gloria Zhao'
>>> > <gloriajzhao@gmail•com>
>>> > Subject: RE: [bitcoin-dev] Package Relay Proposal
>>> >
>>> > > From: bitcoin-dev <bitcoin-dev-bounces@lists•linuxfoundation.org> On
>>> > Behalf
>>> > > Of Anthony Towns via bitcoin-dev
>>> > > Sent: Wednesday, May 25, 2022 11:56 AM
>>> >
>>> > > So the other thing is what happens if the peer announcing packages
>>> to us
>>> > is
>>> > > dishonest?
>>> > >
>>> > > They announce pkg X, say X has parents A B C and the fee rate is
>>> garbage.
>>> > But
>>> > > actually X has parent D and the fee rate is excellent. Do we request
>>> the
>>> > > package from another peer, or every peer, to double check? Otherwise
>>> > we're
>>> > > allowing the first peer we ask about a package to censor that tx from
>>> us?
>>> > >
>>> > > I think the fix for that is just to provide the fee and weight when
>>> > announcing
>>> > > the package rather than only being asked for its info? Then if one
>>> peer
>>> > makes
>>> > > it sound like a good deal you ask for the parent txids from them,
>>> dedupe,
>>> > > request, and verify they were honest about the parents.
>>> >
>>> > Single tx broadcasts do not carry an advertised fee rate, however the'
>>> > feefilter' message (BIP133) provides this distinction. This should be
>>> > interpreted as applicable to packages. Given this message there is no
>>> reason
>>> > to send a (potentially bogus) fee rate with every package. It can only
>>> be
>>> > validated by obtaining the full set of txs, and the only recourse is
>>> > dropping (etc.) the peer, as is the case with single txs. Relying on
>>> the
>>> > existing message is simpler, more consistent, and more efficient.
>>> >
>>> > > >> Is it plausible to add the graph in?
>>> > >
>>> > > Likewise, I think you'd have to have the graph info from many nodes
>>> if
>>> > you're
>>> > > going to make decisions based on it and don't want hostile peers to
>>> be
>>> > able to
>>> > > trick you into ignoring txs.
>>> > >
>>> > > Other idea: what if you encode the parent txs as a short hash of the
>>> wtxid
>>> > > (something like bip152 short ids? perhaps seeded per peer so
>>> collisions
>>> > will
>>> > > be different per peer?) and include that in the inv announcement?
>>> Would
>>> > > that work to avoid a round trip almost all of the time, while still
>>> giving
>>> > you
>>> > > enough info to save bw by deduping parents?
>>> >
>>> > As I suggested earlier, a package is fundamentally a compact block (or
>>> > block) announcement without the header. Compact block (BIP152)
>>> > announcement
>>> > is already well-defined and widely implemented. A node should never be
>>> > required to retain an orphan, and BIP152 ensures this is not required.
>>> >
>>> > Once a validated set of txs within the package has been obtained with
>>> > sufficient fee, a fee-optimal node would accept the largest subgraph of
>>> the
>>> > package that conforms to fee constraints and drop any peer that
>>> provides a
>>> > package for which the full graph does not.
>>> >
>>> > Let us not reinvent the wheel and/or introduce accidental complexity. I
>>> see
>>> > no reason why packaging is not simply BIP152 without the 'header'
>>> field,
>>> an
>>> > updated protocol version, and the following sort of changes to names:
>>> >
>>> > sendpkg
>>> > MSG_CMPCT_PKG
>>> > cmpctpkg
>>> > getpkgtxn
>>> > pkgtxn
>>> >
>>> > > > For a maximum 25 transactions,
>>> > > >23*24/2 = 276, seems like 36 bytes for a child-with-parents package.
>>> > >
>>> > > If you're doing short ids that's maybe 25*4B=100B already, then the
>>> above
>>> > is
>>> > > up to 36% overhead, I guess. Might be worth thinking more about, but
>>> > maybe
>>> > > more interesting with ancestors than just parents.
>>> > >
>>> > > >Also side note, since there are no size/count params,
>>> >
>>> > Size is restricted in the same manner as block and transaction
>>> broadcasts,
>>> > by consensus. If the fee rate is sufficient there would be no reason to
>>> > preclude any valid size up to what can be mined in one block (packaging
>>> > across blocks is not economically rational under the assumption that
>>> one
>>> > miner cannot expect to mine multiple blocks in a row). Count is
>>> incorporated
>>> > into BIP152 as 'shortids_length'.
>>> >
>>> > > > wondering if we
>>> > > >should just have "version" in "sendpackages" be a bit field instead
>>> of
>>> > > >sending a message for each version. 32 versions should be enough
>>> right?
>>> >
>>> > Adding versioning to individual protocols is just a reflection of the
>>> > insufficiency of the initial protocol versioning design, and that of
>>> the
>>> > various ad-hoc changes to it (including yet another approach in this
>>> > proposal) that have been introduced to compensate for it, though I'll
>>> > address this in an independent post at some point.
>>> >
>>> > Best,
>>> > e
>>> >
>>> > > Maybe but a couple of messages per connection doesn't really seem
>>> worth
>>> > > arguing about?
>>> > >
>>> > > Cheers,
>>> > > aj
>>> > >
>>> > >
>>> > > --
>>> > > Sent from my phone.
>>> > > _______________________________________________
>>> > > bitcoin-dev mailing list
>>> > > bitcoin-dev@lists•linuxfoundation.org
>>> > > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>>
>>>
>>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists•linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 24656 bytes --]

 permalink raw reply	[|nested]
2022-05-17 16:01 Gloria Zhao
  2022-05-17 17:56 ` Greg Sanders
  2022-05-18  0:35 ` Anthony Towns

  2022-11-01 18:03   ` Gloria Zhao
  
From: Antoine Riard @ 2022-06-17 20:08 UTC (permalink / raw)
  To: Gloria Zhao, Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 37241 bytes --]

Hi Gloria,

Thanks for working on that,

> Always overestimating fees may sidestep this issue temporarily (while
mempool
> traffic is low and predictable), but this solution is not foolproof
> and wastes users' money. The feerate market can change due to sudden
> spikes in traffic (e.g. huge 12sat/vB dump a few days ago [9]) or
> sustained, high volume of Bitcoin payments (e.g.  April 2021 and
> December 2017).

Even if the LN implementations started to overestimate fees based on the
historical worst-case of block inclusion feerates, there is still room for
exploitation due to bip125 rule#3. Indeed, as long as the adversary is able
to stick in the mempool a higher fee package while the feerate is not
compelling enough to get it mined, your "honest" LN package should be
bounced off.

Considering Core's `MAX_STANDARD_TX_WEIGHT` of 400000 WU, I think it's
practical for an attacker to succeed with this pinning tactic in periods of
traffic spikes. Of course, LN implementation could overestimate fees with a
target like `MAX_STANDARD_WEIGHT` * `worst_case_block_inclusion_feerate` to
mitigate. However, assuming a value of 20sat for the latter, it would
require from any LN user a minimal channel value of 2000000 satoshis to be
theoretically secure against this type of pinning.

So package relay is required to mitigate efficiently and realistically
against pinning attacks, while conserving the same level of "economic"
openness for Lightning. Beyond, it should be also noted that package relay
is only building block of the full set of mitigations, and there should be
a yet to-find-consensus-as-of-today other policy change such as
user-elected package limits or replace-by-feerate.

Anyway, I think it would be beneficial to document the design trade-offs of
pinning mitigations in the `Rationale` subsection, at the attention of
future L2s devs and users ?

> {|
> |  Field Name  ||  Type  ||  Size  ||  Purpose
> |-
> |version || uint32_t || 4 || Denotes a package version supported by the
> node.
> |-
> |max_count || uint32_t || 4 ||Specifies the maximum number of transactions
> per package this node is
> willing to accept.
> |-
> |max_weight || uint32_t || 4 ||Specifies the maximum total weight per
> package this node is willing
> to accept.
> |-
> |}

It's unclear to me what's the purpose of `max_count` and `max_weight` in
the overall package relay flow, if they are intended to be exposed as
configurable settings to node operators. If those fields are present to
allow DoS protection increase of low-performance host, I believe it would
be better to restrain the number of consumed UTXOs or executed sigops per
package, as DoS vectors are more likely to be CPU-based, rather than
memory-based as package size already bounded at acceptance by
`MAX_PACKAGE_COUNT`.

Thinking more we might introduce a `MAX_SIGOPS_PER_PACKAGR` limit, as
otherwise if we naively grant one package announcement as equal to one
transaction announcement in our tx-request logic, we might increase our DoS
surface, node ressources staying equivalent ?

> {|
> |  Field Name  ||  Type  ||  Size  ||   Purpose
> |-
> |txns_length||CompactSize||1 or 3 bytes|| The number of transactions
> requested.

I'm not sure if we'll ever allow 3-bytes of package size, that would be
~32k of transactions.

> |-
> |txns||List of wtxids||txns_length * 32|| The wtxids of each transaction
in
> the package.
> |}

I think there is a bandwidth consumption trade-off to be aware of in the
function of the package-relay usage. Let's consider a single issuer
broadcasting the package to spend a shared-utxo, after the first shot the
parent component should be spread across the network mempools. At each
fee-bump, only the bumped CPFP will propagate on the network, the parent
wtxid is reannounced in `pckginfo1` though there is no need to fetch it
redundantly and waste bandwidth.

However, I think the bandwidth saving does not hold in case of competing
transaction issuers to spend a shared-utxo. In that case, the parent might
differ at each broadcast and the list of wtxid is dissemblable at every
claim of the shared-utxo. We could save the 32 bytes * number of packages
elements by announcing a package_id, computed from the list of wtxids.

I don't know about the occurrence of competing broadcasts among LN
non-cooperative closes, where bandwidth could be potentially saved. I would
say it's likely low because IIRC there is nothing in the LN protocol where
the counterparties signal to each other they're going on-chain to introduce
a competing broadcast synchronizing event. That said, it might increase in
the future in a post-eltoo, multi-party contracting protocol world.

So it might be interesting to document this design trade-off, if we seek
bandwidth optimizations in function of a changing landscape in the type of
transaction issuers in the future.

> 3. The sender provides package information using "pckginfo1",
>    including the blockhash of the sender's best block, the wtxids of
> the transactions in the package, their total fees and total weight.

It's unclear to me how the `pckinfo1` receiver should proceed if the
sender's best block is not in sync with the local chain tip.

If the package isn't processed further, that's annoying for all the
low-performance  LN mobile clients, their chain tips might be always behind
by few blocks from the p2p network nodes. It sounds like their packages
won't propagate at all.

If the package is processed further whatever the sender-receiver sync on
chain tip, what's the purpose of including the blockhash ?

> A child-with-unconfirmed-parents package for a transaction should be
> announced when it meets the peer's fee filter but one or more of its
> parents don't; a "inv(MSG_PCKG1)" instead of "inv(WTX)" should be sent
> for the child. Each of the parents which meet the peer's fee filter
> should still be announced normally.

I believe we might have concerns of package-feerate downgrades attacks.
E.g, in the LN context, where your channel counterparty is aiming to jam
the propagation of the best-feerate version of the package.

Let's say you have :
- Alice's commitment_tx, at 1s/vB
- package A + child B, at 3s/vB
- package A + child C, at 10s/vB
- block inclusion feerate at 10s/vB
- Alice and Mallory are LN channel counterparties
- commitment_tx is using LN's anchor outputs

Alice's LN node broadcasts A+C to her mempool.
Bob's feefilter is at 3s/vB.
Mallory broadcasts her child B in Alice's mempool.
LN commitment does not meet Bob's feefilter.
Package A+child B at 3s/vB meets Bob's feefilter and is announced to Bob.
Mallory broadcasts her own commitment_tx at 4s/vB in Bob's mempool.
When Alice's child C is relayed to Bob, it's bounced off Bob's mempool.

Do you think this situation is plausible ? Of course, it might be heavily
dependent on package-relay yet-not-implemented internal p2p logic.
I think it could be fixable if LN removes the counterparty's
`anchor_output` on the local node's version of the commitment transaction,
once package relay is deployed.

Another question, at the next fee-bump iteration, Alice rebroadcasts
A+child D, at 12 s/vB. Her node has already marked Alice's commitment_tx as
known in Bob's `m_tx_inventory_known_filter`. So when a new higher fee
child is  discovered, should a `child-with-unconfirmed-parents` be
announced between Alice and Bob ?

Anyway, I think it would be interesting to pseudo-specify the
package-assemblage algorithm (or if there is code already available) to see
if it's robust against adversarial or unlucky situations ?

> In fact, a package
> of transactions may be announced using both Erlay and package relay.
> After reconciliation, if the initiator would have announced a
> transaction by wtxid but also has package information for it, they may
> send "inv(MSG_PCKG)" instead of "inv(WTX)".

Yes, I think this holds. Note, we might have to add to the reconciliation
set low-fee parents succeeding the feefilter check due to a child. When the
reconcildiff, we might have to bifucarte again on feefilter to decide to
announce missing wtixds either as `inv(MSG_PCKG)` or `inv(WTX)`.

(IIRC, I've already made few feedbacks offline though good to get them in
the public space and think more)

Antoine

Le mar. 17 mai 2022 à 12:09, Gloria Zhao via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> a écrit :

> Hi everybody,
>
> I’m writing to propose a set of p2p protocol changes to enable package
> relay, soliciting feedback on the design and approach. Here is a link
> to the most up-to-date proposal:
>
> https://github.com/bitcoin/bips/pull/1324
>
> If you have concept or approach feedback, *please respond on the
> mailing list* to allow everybody to view and participate in the
> discussion. If you find a typo or inaccurate wording, please feel free
> to leave suggestions on the PR.
>
> I’m also working on an implementation for Bitcoin Core.
>
>
> The rest of this post will include the same contents as the proposal,
> with a bit of reordering and additional context. If you are not 100%
> up-to-date on package relay and find the proposal hard to follow, I
> hope you find this format more informative and persuasive.
>
>
> ==Background and Motivation==
>
> Users may create and broadcast transactions that depend upon, i.e.
> spend outputs of, unconfirmed transactions. A “package” is the
> widely-used term for a group of transactions representable by a
> connected Directed Acyclic Graph (where a directed edge exists between
> a transaction that spends the output of another transaction).
>
> Incentive-compatible mempool and miner policies help create a fair,
> fee-based market for block space. While miners maximize transaction
> fees in order to earn higher block rewards, non-mining users
> participating in transaction relay reap many benefits from employing
> policies that result in a mempool with the same contents, including
> faster compact block relay and more accurate fee estimation.
> Additionally, users may take advantage of mempool and miner policy to
> bump the priority of their transactions by attaching high-fee
> descendants (Child Pays for Parent or CPFP).  Only considering
> transactions one at a time for submission to the mempool creates a
> limitation in the node's ability to determine which transactions have
> the highest feerates, since it cannot take into account descendants
> until all the transactions are in the mempool. Similarly, it cannot
> use a transaction's descendants when considering which of two
> conflicting transactions to keep (Replace by Fee or RBF).
>
> When a user's transaction does not meet a mempool's minimum feerate
> and they cannot create a replacement transaction directly, their
> transaction will simply be rejected by this mempool. They also cannot
> attach a descendant to pay for replacing a conflicting transaction.
> This limitation harms users' ability to fee-bump their transactions.
> Further, it presents a security issue in contracting protocols which
> rely on **presigned**, time-sensitive transactions to prevent cheating
> (HTLC-Timeout in LN Penalty [1] [2] [3], Unvault Cancel in Revault
> [4], Refund Transaction in Discreet Log Contracts [5], Updates in
> eltoo [6]). In other words, a key security assumption of many
> contracting protocols is that all parties can propagate and confirm
> transactions in a timely manner.
>
> In the past few years, increasing attention [0][1][2][3][6] has been
> brought to **pinning attacks**, a type of censorship in which the
> attacker uses mempool policy restrictions to prevent a transaction
> from being relayed or getting mined.  TLDR: revocation transactions
> must meet a certain confirmation target to be effective, but their
> feerates are negotiated well ahead of broadcast time. If the
> forecasted feerate was too low and no fee-bumping options are
> available, attackers can steal money from their counterparties. I walk
> through a concrete example for stealing Lightning HTLC outputs at
> ~23:58 in this talk [7][8].  Note that most attacks are only possible
> when the market for blockspace at broadcast time  demands much higher
> feerates than originally anticipated at signing time. Always
> overestimating fees may sidestep this issue temporarily (while mempool
> traffic is low and predictable), but this solution is not foolproof
> and wastes users' money. The feerate market can change due to sudden
> spikes in traffic (e.g. huge 12sat/vB dump a few days ago [9]) or
> sustained, high volume of Bitcoin payments (e.g.  April 2021 and
> December 2017).
>
> The best solution is to enable nodes to consider packages of
> transactions as a unit, e.g. one or more low-fee parent transactions
> with a high-fee child, instead of separately. A package-aware mempool
> policy can help determine if it would actually be economically
> rational to accept a transaction to the mempool if it doesn't meet fee
> requirements individually. Network-wide adoption of these policies
> would create a more purely-feerate-based market for block space and
> allow contracting protocols to adjust fees (and therefore mining
> priority) at broadcast time.  Some support for packages has existed in
> Bitcoin Core for years. Since v0.13, Bitcoin Core has used ancestor
> packages instead of individual transactions to evaluate the incentive
> compatibility of transactions in the mempool [10] and select them for
> inclusion in blocks [11].
>
> Package Relay, the concept of {announcing, requesting, downloading}
> packages between nodes on the p2p network, has also been discussed for
> many years. The earliest public mention I can find is from 2015 [12].
> The two most common use cases for package relay are fee-bumping
> otherwise-too-low-fee transactions and reducing the amount of orphans.
> It seems uncontroversial to say that everybody desires package relay
> conceptually, with varying degrees of urgency. Lots of work has been
> done by others over the past few years, from which I've taken
> inspiration from [13][14][15][16].
>
> My approach has been to split the project into two components: (1) Package
> Mempool Accept, which includes validation logic and mempool policy.
> (3) Package Relay, which includes the p2p protocol changes.
>
> Progress so far:
> After discussions with various developers of contracting protocols
> (with heavier emphasis towards LN), it was determined that a
> package containing a child with all of its unconfirmed parents
> (child-with-unconfirmed-parents or 1-child-multi-parent package) would
> be sufficient for their use case, i.e. fee-bumping presigned
> transactions. A child-with-unconfirmed-parents package has several
> properties that make many things easier to reason about.
>
> A few months ago, I proposed a set of policies for safe package
> validation and fee assessment for packages of this restricted
> topology [17]. A series of PRs implementing this proposal have
> been merged into Bitcoin Core [18].
>
> Theoretically, developing a safe and incentive-compatible package
> mempool acceptance policy is sufficient to solve this issue. Nodes
> could opportunistically accept packages (e.g. by trying combinations
> of transactions rejected from their mempools), but this practice would
> likely be inefficient at best and open new Denial of Service attacks
> at worst. Additional p2p messages may enable nodes to request and
> share package validation-related information with one another in a
> more communication-efficient way.
>
> Given that only package RBF remains for package mempool accept, and we
> can make progress on p2p and mempool in parallel, I think it’s
> appropriate to put forward a package relay proposal.
>
> ==Proposal==
>
> This proposal contains 2 components: a “generic” package relay
> protocol and an extension of it, child-with-unconfirmed-parents
> packages, as version 1 package relay. Another version of packages,
> “tx-with-unconfirmed-ancestors” can be created to extend package relay
> for eliminating orphans.
>
> ===Generic Package Relay===
>
> Two main ideas are introduced:
>
> Download and validate packages of transactions together.
>
> Provide information to help peers decide whether to request and/or how
> to validate transactions which are part of a package.
>
> ====Intended Protocol Flow====
>
> Due to the asynchronous nature of a distributed transaction relay
> network, nodes may not receive all of the information needed to
> validate a transaction at once. For example, after a node completes
> Initial Block Download (IBD) and first starts participating in
> transaction relay with an empty mempool, it is common to receive
> orphans. In such scenarios where a node is aware that it is missing
> information, a ''receiver-initiated'' dialogue is appropriate:
>
> 1. Receiver requests package information.
>
> 2. The sender provides package information, including the wtxids of
>    the transactions in the package and anything else that might be
> relevant (e.g. total fees and size).
>
> 3. The reciever uses the package information to decide how to request
>    and validate the transactions.
>
> Sometimes, no matter what order transactions are received by a node,
> validating them individually is insufficient. When the sender is aware
> of additional information that the receiver needs to accept a package,
> a proactive ''sender-initiated'' dialogue should be enabled:
>
> 1. Sender announces they have package information pertaining to a
>    transaction that might otherwise be undesired on its own.
>
> 2. The receiver requests package information.
>
> 3. The sender provides package information, including the wtxids of
>    the transactions in the package and anything else that might be
> relevant (e.g. total fees and size).
>
> 4. The reciever uses the package information to decide how to request
>    and validate the transactions.
>
> Package relay is negotiated between two peers during the version
> handshake. Package relay requires both peers to support wtxid-based
> relay because package transactions are referenced by their wtxid.
>
> ====New Messages====
>
> Three new protocol messages are added for use in any version of
> package relay. Additionally, each version of package relay must define
> its own inv type and "pckginfo" message version, referred to in this
> document as "MSG_PCKG" and "pckginfo" respectively. See
> BIP-v1-packages for a concrete example.
>
> =====sendpackages=====
>
> {|
> |  Field Name  ||  Type  ||  Size  ||  Purpose
> |-
> |version || uint32_t || 4 || Denotes a package version supported by the
> node.
> |-
> |max_count || uint32_t || 4 ||Specifies the maximum number of transactions
> per package this node is
> willing to accept.
> |-
> |max_weight || uint32_t || 4 ||Specifies the maximum total weight per
> package this node is willing
> to accept.
> |-
> |}
>
> 1. The "sendpackages" message has the structure defined above, with
>    pchCommand == "sendpackages".
>
> 2. During version handshake, nodes should send a "sendpackages"
>    message indicate they support package relay and may request
> packages.
>
> 3. The message should contain a version supported by the node. Nodes
>    should send a "sendpackages" message for each version they support.
>
> 4. The "sendpackages" message MUST be sent before sending a "verack"
>    message. If a "sendpackages" message is received afer "verack", the
> sender should be disconnected.
>
> 5. If 'fRelay==false' in a peer's version message, the node must not
>    send "sendpackages" to them. If a "sendpackages" message is
> received by a peer after sending `fRelay==false` in their version
> message, the sender should be disconnected.
>
> 6.. Upon receipt of a "sendpackages" message with a version that is
> not supported, a node must treat the peer as if it never received the
> message.
>
> 7. If both peers send "wtxidrelay" and "sendpackages" with the same
>    version, the peers should announce, request, and send package
> information to each other.
>
> =====getpckgtxns=====
>
> {|
> |  Field Name  ||  Type  ||  Size  ||   Purpose
> |-
> |txns_length||CompactSize||1 or 3 bytes|| The number of transactions
> requested.
> |-
> |txns||List of wtxids||txns_length * 32|| The wtxids of each transaction
> in the package.
> |}
>
> 1. The "getpckgtxns" message has the structure defined above, with
>    pchCommand == "getpckgtxns".
>
> 2. A "getpckgtxns" message should be used to request all or some of
>    the transactions previously announced in a "pckginfo" message,
> specified by witness transactiosome id.
>
> 3. Upon receipt of a "getpckgtxns" message, a node must respond with
>    either a "pckgtxns" containing the requested transactions or a
> "notfound" message indicating one or more of the transactions is
> unavailable. This allows the receiver to avoid downloading and storing
> transactions that cannot be validated immediately.
>
> 4. A "getpckgtxns" message should only be sent if both peers agreed to
>    send packages in the version handshake. If a "getpckgtxns" message
> is received from a peer with which package relay was not negotiated,
> the sender should be disconnected.
>
> =====pckgtxns=====
>
> {|
> |  Field Name  ||  Type  ||  Size  ||   Purpose
> |-
> |txns_length||CompactSize||1 or 3 bytes|| The number of transactions
> provided.
> |-
> |txns||List of transactions||variable|| The transactions in the package.
> |}
>
> 1. The "pckgtxns" message has the structure defined above, with
>    pchCommand == "pckgtxns".
>
> 2. A "pckgtxns" message should contain the transaction data requested
>    using "getpckgtxns".
>
> 3. A "pckgtxns" message should only be sent to a peer that requested
>    the package using "getpckgtxns". If a node receives an unsolicited
> package, the sender should be disconnected.
>
> 4. A "pckgtxns" message should only be sent if both peers agreed to
>    send packages in the version handshake. If a "pckgtxns" message is
> received from a peer with which package relay was not negotiated, the
> sender should be disconnected.
>
> ===Version 1 Packages: child-with-unconfirmed-parents===
>
> This extends package relay for packages consisting of one transaction
> and all of its unconfirmed parents,by defining version 1 packages, a
> pckginfo1 message, and a MSG_PCKG1 inv type. It enables the use case
> in which a child pays for its otherwise-too-low-fee parents and their
> mempool conflict(s).
>
> ====Intended Protocol Flow====
>
> When relaying a package of low-fee parent(s) and high-fee child, the
> sender and receiver do the following:
>
> 1. Sender announces they have a child-with-unconfirmed-parents package
>    for a child that pays for otherwise-too-low-fee parent(s) using
> "inv(MSG_PCKG1)".
>
> 2. The receiver requests package information using
>    "getdata(MSG_PCKG1)".
>
> 3. The sender provides package information using "pckginfo1",
>    including the blockhash of the sender's best block, the wtxids of
> the transactions in the package, their total fees and total weight.
>
> 4. The reciever uses the package information to decide how to request
>    the transactions. For example, if the receiver already has some of
> the transactions in their mempool, they only request the missing ones.
> They could also decide not to request the package at all based on the
> fee information provided.
>
> 5. Upon receiving a "pckgtxns", the receiver submits the transactions
>    together as a package.
>
> ====New Messages====
>
> A new inv type, "MSG_PCKG1", and new protocol message, "PCKGINFO1",
> are added.
>
> =====pckginfo1=====
>
> {|
> |  Field Name  ||  Type  ||  Size  ||   Purpose
> |-
> |blockhash || uint256 || 32 || The chain tip at which this package is
> defined.
> |-
> |pckg_fee||CAmount||4|| The sum total fees paid by all transactions in the
> package.
> |-
> |pckg_weight||int64_t||8|| The sum total weight of all transactions in the
> package.
> |-
> |txns_length||CompactSize||1 or 3 bytes|| The number of transactions
> provided.
> |-
> |txns||List of wtxids||txns_length * 32|| The wtxids of each transaction
> in the package.
> |}
>
>
> 1. The "pckginfo1" message has the structure defined above, with
>    pchCommand == "pckginfo1".
>
> 2. A "pckginfo1" message contains information about a version 1
>    package (defined below), referenced by the wtxid of the transaction
> it pertains to and the current blockhash.
>
> 3. Upon receipt of a "pckginfo1" message, the node should decide if it
>    wants to validate the package, request transaction data if
> necessary, etc.
>
> 4. Upon receipt of a malformed "pckginfo1" message or package that
>    does not abide by the max_count, max_weight, or other rules
> specified by the version agreed upon in the initial negotiation, the
> sender should be disconnected.  If a node receives a "pckginfo1"
> message for which the "pckg_fee" or "pckg_weight" do not reflect the
> true total fees and weight, respectively, or the transactions in the
> package, the message is malformed.
>
> 5. A node MUST NOT send a "pckginfo1" message that has not been
>    requested by the recipient. Upon receipt of an unsolicited
> "pckginfo1", a node should disconnect the sender.
>
> 6. A "pckginfo1" message should only be sent if both peers agreed to
>    send version 1 packages in the version handshake. If a "pckginfo1"
> message is received from a peer with which package relay was not
> negotiated, the sender should be disconnected.
>
> =====MSG_PCKG1=====
>
> 1. A new inv type (MSG_PCKG1 == 0x6) is added, for use in inv messages
>    and getdata requests pertaining to version 1 packages.
>
> 2. As an inv type, it indicates that both transaction data and version
>    1 package information are available for the transaction. The
> transaction is referenced by its wtxid. As a getdata request type, it
> indicates that the sender wants package information for the
> transaction.
>
> 3. Upon receipt of a "getdata" request for "MSG_PCKG1", the node
>    should respond with the version 1 package corresponding to the
> requested transaction and its current chain tip, or with NOTFOUND.
> The node should not assume that the sender is requesting the
> transaction data as well.
>
> ====Child With Parent Packages Rules====
>
> A child-with-unconfirmed-parents package sent between nodes must abide
> by the rules below, otherwise the package is malformed and the sender
> should be disconnected.
>
> A version 1 or ''child-with-unconfirmed-parents'' package can be
> defined for any transaction that spends unconfirmed inputs. The child
> can be thought of as the "representative" of the package. This package
> can be uniquely identified by the transaction's wtxid and the current
> chain tip block hash.
>
> A ''child-with-unconfirmed-parents'' package MUST be:
>
> 1. ''Sorted topologically.'' For every transaction t in the package,
>    if any of t's parents are present in the package, the parent must
> appear somewhere in the list before t. In other words, the
> transactions must be sorted in ascending order of the number of
> ancestors present in the package.
>
> 2. ''Only 1 child with unconfirmed parents.'' The package must consist
>    of one transaction and its unconfirmed parents. There must not be
> any other transactions in the package. Other dependency relationships
> may exist within the package (e.g. one parent may spend the output of
> another parent) provided that topological order is respected.
>
> 3. ''All unconfirmed parents.'' All of the child's unconfirmed parents
>    must be present.
>
> 4. ''No conflicts.'' None of the transactions in the package may
>    conflict with each other (i.e.  spend the same prevout).
>
> 5. ''Total fees and weight.'' The 'total_fee' and 'total_weight'
>    fields must accurately represent the sum total of all transactions'
> fees and weights as defined in BIP141, respectively.
>
> Not all of the child's parents must be present; the child transaction
> may also spend confirmed inputs. However, if the child has confirmed
> parents, they must not be in the package.
>
> While a child-with-unconfirmed-parents package is perhaps most
> relevant when the child has a higher feerate than its parents, this
> property is not required to construct a valid package.
>
> ====Clarifications====
>
> ''Q: Under what circumstances should a sender announce a
> child-with-unconfirmed-parents package?''
>
> A child-with-unconfirmed-parents package for a transaction should be
> announced when it meets the peer's fee filter but one or more of its
> parents don't; a "inv(MSG_PCKG1)" instead of "inv(WTX)" should be sent
> for the child. Each of the parents which meet the peer's fee filter
> should still be announced normally.
>
> ''Q: What if a new block arrives in between messages?''
>
> A child-with-unconfirmed-parents package is defined for a transaction
> based on the current chain state. As such, a new block extending the
> tip may decrease the number of transactions in the package (i.e. if
> any of the transaction's parents were included in the block). In a
> reorg, the number of transactions in the package may decrease or
> increase (i.e. if any of the transaction's parents were included in a
> block in the previous chain but not the new one).
>
> If the new block arrives before the "getdata" or "pckginfo1", nothing
> needs to change.
>
> If the new block arrives before "getpckgtxns" or before "pckgtxns",
> the receiver may need to re-request package information if the block
> contained a transaction in the package. If the block doesn't contain
> any transactions in the package, whether it extends the previous tip
> or causes a reorg, nothing needs to change.
>
> ''Q: Can "getpckgtxns" and "pckgtxns" messages contain only one
> transaction?''
>
> Yes.
>
> ===Further Protocol Extensions===
>
> When introducing a new type of package, assign it a version number "n"
> and use an additional "sendpackages" message during version handshake
> to negotiate support for it. An additional package information message
> "pckginfon" and inv type "MSG_PCKGn" should be defined for the type of
> package.  However, "getpckgtxns" and "pckgtxns" do not need to be
> changed.
>
> Example proposal for tx-with-unconfirmed-ancestors package relay: [19]
>
> ===Compatibility===
>
> Older clients remain fully compatible and interoperable after this
> change. Clients implementing this protocol will only attempt to send
> and request packages if agreed upon during the version handshake.
>
> ===Package Erlay===
>
> Clients using BIP330 reconciliation-based transaction relay (Erlay)
> are able to use package relay without interference. In fact, a package
> of transactions may be announced using both Erlay and package relay.
> After reconciliation, if the initiator would have announced a
> transaction by wtxid but also has package information for it, they may
> send "inv(MSG_PCKG)" instead of "inv(WTX)".
>
> ===Rationale===
>
> ====P2P Message Design====
>
> These p2p messages are added for communication efficiency and, as
> such, one should measure alternative solutions based on the resources
> used to communicate (not necessarily trustworthy) information: We
> would like to minimize network bandwidth, avoid downloading a
> transaction more than once, avoid downloading transactions that are
> eventually rejected, and minimize storage allocated for
> not-yet-validated transactions.
>
> Consider these (plausible) scenarios in transaction relay:
>
> Alice (the "sender") is relaying transactions to Bob (the "receiver").
> Alice's mempool has a minimum feerate of 1sat/vB and Bob's has a
> minimum feerate of 3sat/vB. For simplicity, all transactions are
> 1600Wu in virtual size and 500 bytes in serialized size. Apart from
> the spending relationships specified, all other inputs are from
> confirmed UTXOs.
>
> 1. Package {A, B} where A pays 0 satoshis and B pays 8000 satoshis in
>    fees.
>
> 2. Package {C, D} where C pays 0 satoshis and D pays 1200 satoshis in
>    fees.
>
> 3. Package {E, F, G, H, J} that pays 4000, 8000, 0, 2000, and 4000
>    satoshis in fees, respectively.
>
> ====Alternative Designs Considered====
>
> ''Package Information Only:'' Just having "pckginfo" gives enough
> information for the receiver to accept the package. Omit the
> "getpckgtxns" and "pckgtxns" messages. While this option is a good
> fallback if batched transaction download fails for some reason, it
> shouldn't be used as the default because it 'always' requires storage
> of unvalidated transactions.
>
> ''No Package Information Round:'' Instead of having a package
> information round, just use the child's wtxid to refer to the package
> and always send the entire package together. This would cause nodes to
> redownload duplicate transactions.
>
> I have also created a slidedeck exploring various alternative designs
> and some examples in which they fall flat [20]. Please feel free to
> suggest other alternatives.
>
> ====Versioning System====
>
> This protocol should be extensible to support multiple types of
> packages based on future desired use cases. Two "flavors" of
> versioning were considered:
>
> 1. When package mempool acceptance is upgraded to support more types
>    of packages, increment the version number (similar to Erlay).
> During version handshake, peers negotiate which version of package
> relay they will use by each sending one "sendpackages" message.
>
> 2. When introducing another type of package, assign a version number
>    to it and announce it as an additional supported version (similar
> to Compact Block Relay). During version handshake, peers send one
> "sendpackages" message for each version supported.
>
> The second option was favored because it allows different parameters
> for different versions.  For example, it should be possible to support
> both "arbitrary topology but maximum 3-transaction" package as well as
> "child-with-unconfirmed-parents with default mempool ancestor limits"
> packages simultaneously.
>
> ==Acknowledgements==
>
> I hope to have made it abundantly clear that this proposal isn’t
> inventing the concept of package relay, and in fact builds upon years
> of work by many others, including Suhas Daftuar and Antoine Riard.
>
> Thank you to John Newbery and Martin Zumsande for input on the design.
>
> Thank you to Matt Corallo, Christian Decker, David Harding, Antoine
> Poinsot, Antoine Riard, Gregory Sanders, Chris Stewart, Bastien
> Teinturier, and others for input on the desired interface for
> contracting protocols.
>
> Looking forward to hearing your thoughts!
>
> Best,
> Gloria
>
> [0]:
> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-January/019817.html
> [1]:
> https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-April/002639.html
> [2]:
> https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-June/002758.html
> [3]:
> https://github.com/t-bast/lightning-docs/blob/master/pinning-attacks.md
> [4]:
> https://github.com/revault/practical-revault/blob/master/transactions.md#cancel_tx
> [5]:
> https://github.com/discreetlogcontracts/dlcspecs/blob/master/Transactions.md#refund-transaction
> [6]: https://gist.github.com/instagibbs/60264606e181451e977e439a49f69fe1
> [7]:
> https://btctranscripts.com/adopting-bitcoin/2021/2021-11-16-gloria-zhao-transaction-relay-policy/#lightning-attacks
> [8]: https://youtu.be/fbWSQvJjKFs?t=1438
> [9]:
> https://www.reddit.com/r/Bitcoin/comments/unew4e/looks_like_70_mvb_of_transactions_just_got_dumped/
> [10]: https://github.com/bitcoin/bitcoin/pull/7594
> [11]: https://github.com/bitcoin/bitcoin/pull/7600
> [12]: https://github.com/bitcoin/bitcoin/pull/6455#issuecomment-122716820
> [13]: https://gist.github.com/sdaftuar/8756699bfcad4d3806ba9f3396d4e66a
> [14]: https://github.com/bitcoin/bitcoin/issues/14895
> [15]: https://github.com/bitcoin/bitcoin/pull/16401
> [16]: https://github.com/bitcoin/bitcoin/pull/19621
> [17]:
> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-September/019464.html
> [18]: https://github.com/users/glozow/projects/5/views/4?layout=board
> [19]: https://gist.github.com/glozow/9b321cd3ef6505135c763112033ff2a7
> [20]:
> https://docs.google.com/presentation/d/1B__KlZO1VzxJGx-0DYChlWawaEmGJ9EGApEzrHqZpQc/edit?usp=sharing
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 41950 bytes --]

 permalink raw reply	[|nested]
2022-06-17 20:08 ` Antoine Riard

  
From: Gloria Zhao @ 2022-11-01 18:03 UTC (permalink / raw)
  To: Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 41516 bytes --]

Hi everyone,

I've made some significant changes to my package relay proposal based on
observations while implementing, feedback on this thread, and offline
discussions [1].

The new proposal is called Ancestor Package Relay, BIP331, and PR'd at
https://github.com/bitcoin/bips/pull/1382

The major changes to the proposal are:
1. Scope reduction to receiver-initiated only
2. Scope reduction to ancestor packages only
3. Removal of block hash from package information

1.  Scope reduction to receiver-initiated only
Receiver-intiated package relay enables a node to ask for more information
when they suspect they are missing something (i.e. in this case to resolve
a missing parent tx). Sender-initiated package relay should, theoretically,
save a round trip by notifying the receiver ahead of time that "hey, this
is going to be a package, so make sure you download and submit these
transactions together." As with any proactive communication, there is a
chance that the node already knows this information, so this network
bandwidth was wasted. The logic used to decide _when_ to announce a package
proactively determines whether it is a net increase or decrease for overall
bandwidth usage. However, it's difficult to design anything to save
bandwidth without any idea of what its bandwidth usage actually looks like
in practice. We'll want to design the sender-initiated protocol carefully,
and inform the design decisions using data collected from the mainnet p2p
network. However, there is no historical transaction data to use because
the goal is to enable currently-rejected transactions to propagate. In
order to get this right, I propose we hold off on sender-initiated for now,
deploy receiver-initiated package relay, observe its usage and figure out
where we can save a round trip, and then produce a well-researched
sender-initiated package relay proposal.

2. Scope reduction to ancestor packages only
The proposal now only includes ancestor packages (previously called
tx-with-unconfirmed-ancestors or "v2" packages). The
child-with-unconfirmed-parents (previously called "v1") package has been
removed since it is a subset of ancestor packages and sender-initiated
relay has been removed. It may be relevant again in the future with
sender-initiated packages. If you were reviewing the previous proposal,
"pkginfo2" message has been renamed to "ancpkginfo" and "MSG_PKGINFO2" inv
type to "MSG_ANCPKGINFO".

3. Removal of block hash from package information
Most of the rationale is already on this thread. The block hash was an
attempt to enforce topology when chainstates differ, but isn't worth it. It
does not make much sense to drop or delay transaction data requests due to
mismatched chainstates, and the chainstate may change again between package
information and transaction data rounds. Instead, differences in chainstate
should be handled internally at the mempool validation level. The node
should de-duplicate recently-confirmed transactions and make a best effort
to validate the transactions it has already downloaded.

Thanks,
Gloria

[1]
https://diyhpl.us/wiki/transcripts/bitcoin-core-dev-tech/2022-10-11-package-relay/

On Fri, Jun 17, 2022 at 9:08 PM Antoine Riard <antoine.riard@gmail•com>
wrote:

> Hi Gloria,
>
> Thanks for working on that,
>
> > Always overestimating fees may sidestep this issue temporarily (while
> mempool
> > traffic is low and predictable), but this solution is not foolproof
> > and wastes users' money. The feerate market can change due to sudden
> > spikes in traffic (e.g. huge 12sat/vB dump a few days ago [9]) or
> > sustained, high volume of Bitcoin payments (e.g.  April 2021 and
> > December 2017).
>
> Even if the LN implementations started to overestimate fees based on the
> historical worst-case of block inclusion feerates, there is still room for
> exploitation due to bip125 rule#3. Indeed, as long as the adversary is able
> to stick in the mempool a higher fee package while the feerate is not
> compelling enough to get it mined, your "honest" LN package should be
> bounced off.
>
> Considering Core's `MAX_STANDARD_TX_WEIGHT` of 400000 WU, I think it's
> practical for an attacker to succeed with this pinning tactic in periods of
> traffic spikes. Of course, LN implementation could overestimate fees with a
> target like `MAX_STANDARD_WEIGHT` * `worst_case_block_inclusion_feerate` to
> mitigate. However, assuming a value of 20sat for the latter, it would
> require from any LN user a minimal channel value of 2000000 satoshis to be
> theoretically secure against this type of pinning.
>
> So package relay is required to mitigate efficiently and realistically
> against pinning attacks, while conserving the same level of "economic"
> openness for Lightning. Beyond, it should be also noted that package relay
> is only building block of the full set of mitigations, and there should be
> a yet to-find-consensus-as-of-today other policy change such as
> user-elected package limits or replace-by-feerate.
>
> Anyway, I think it would be beneficial to document the design trade-offs
> of pinning mitigations in the `Rationale` subsection, at the attention of
> future L2s devs and users ?
>
> > {|
> > |  Field Name  ||  Type  ||  Size  ||  Purpose
> > |-
> > |version || uint32_t || 4 || Denotes a package version supported by the
> > node.
> > |-
> > |max_count || uint32_t || 4 ||Specifies the maximum number of
> transactions
> > per package this node is
> > willing to accept.
> > |-
> > |max_weight || uint32_t || 4 ||Specifies the maximum total weight per
> > package this node is willing
> > to accept.
> > |-
> > |}
>
> It's unclear to me what's the purpose of `max_count` and `max_weight` in
> the overall package relay flow, if they are intended to be exposed as
> configurable settings to node operators. If those fields are present to
> allow DoS protection increase of low-performance host, I believe it would
> be better to restrain the number of consumed UTXOs or executed sigops per
> package, as DoS vectors are more likely to be CPU-based, rather than
> memory-based as package size already bounded at acceptance by
> `MAX_PACKAGE_COUNT`.
>
> Thinking more we might introduce a `MAX_SIGOPS_PER_PACKAGR` limit, as
> otherwise if we naively grant one package announcement as equal to one
> transaction announcement in our tx-request logic, we might increase our DoS
> surface, node ressources staying equivalent ?
>
> > {|
> > |  Field Name  ||  Type  ||  Size  ||   Purpose
> > |-
> > |txns_length||CompactSize||1 or 3 bytes|| The number of transactions
> > requested.
>
> I'm not sure if we'll ever allow 3-bytes of package size, that would be
> ~32k of transactions.
>
> > |-
> > |txns||List of wtxids||txns_length * 32|| The wtxids of each transaction
> in
> > the package.
> > |}
>
> I think there is a bandwidth consumption trade-off to be aware of in the
> function of the package-relay usage. Let's consider a single issuer
> broadcasting the package to spend a shared-utxo, after the first shot the
> parent component should be spread across the network mempools. At each
> fee-bump, only the bumped CPFP will propagate on the network, the parent
> wtxid is reannounced in `pckginfo1` though there is no need to fetch it
> redundantly and waste bandwidth.
>
> However, I think the bandwidth saving does not hold in case of competing
> transaction issuers to spend a shared-utxo. In that case, the parent might
> differ at each broadcast and the list of wtxid is dissemblable at every
> claim of the shared-utxo. We could save the 32 bytes * number of packages
> elements by announcing a package_id, computed from the list of wtxids.
>
> I don't know about the occurrence of competing broadcasts among LN
> non-cooperative closes, where bandwidth could be potentially saved. I would
> say it's likely low because IIRC there is nothing in the LN protocol where
> the counterparties signal to each other they're going on-chain to introduce
> a competing broadcast synchronizing event. That said, it might increase in
> the future in a post-eltoo, multi-party contracting protocol world.
>
> So it might be interesting to document this design trade-off, if we seek
> bandwidth optimizations in function of a changing landscape in the type of
> transaction issuers in the future.
>
> > 3. The sender provides package information using "pckginfo1",
> >    including the blockhash of the sender's best block, the wtxids of
> > the transactions in the package, their total fees and total weight.
>
> It's unclear to me how the `pckinfo1` receiver should proceed if the
> sender's best block is not in sync with the local chain tip.
>
> If the package isn't processed further, that's annoying for all the
> low-performance  LN mobile clients, their chain tips might be always behind
> by few blocks from the p2p network nodes. It sounds like their packages
> won't propagate at all.
>
> If the package is processed further whatever the sender-receiver sync on
> chain tip, what's the purpose of including the blockhash ?
>
> > A child-with-unconfirmed-parents package for a transaction should be
> > announced when it meets the peer's fee filter but one or more of its
> > parents don't; a "inv(MSG_PCKG1)" instead of "inv(WTX)" should be sent
> > for the child. Each of the parents which meet the peer's fee filter
> > should still be announced normally.
>
> I believe we might have concerns of package-feerate downgrades attacks.
> E.g, in the LN context, where your channel counterparty is aiming to jam
> the propagation of the best-feerate version of the package.
>
> Let's say you have :
> - Alice's commitment_tx, at 1s/vB
> - package A + child B, at 3s/vB
> - package A + child C, at 10s/vB
> - block inclusion feerate at 10s/vB
> - Alice and Mallory are LN channel counterparties
> - commitment_tx is using LN's anchor outputs
>
> Alice's LN node broadcasts A+C to her mempool.
> Bob's feefilter is at 3s/vB.
> Mallory broadcasts her child B in Alice's mempool.
> LN commitment does not meet Bob's feefilter.
> Package A+child B at 3s/vB meets Bob's feefilter and is announced to Bob.
> Mallory broadcasts her own commitment_tx at 4s/vB in Bob's mempool.
> When Alice's child C is relayed to Bob, it's bounced off Bob's mempool.
>
> Do you think this situation is plausible ? Of course, it might be heavily
> dependent on package-relay yet-not-implemented internal p2p logic.
> I think it could be fixable if LN removes the counterparty's
> `anchor_output` on the local node's version of the commitment transaction,
> once package relay is deployed.
>
> Another question, at the next fee-bump iteration, Alice rebroadcasts
> A+child D, at 12 s/vB. Her node has already marked Alice's commitment_tx as
> known in Bob's `m_tx_inventory_known_filter`. So when a new higher fee
> child is  discovered, should a `child-with-unconfirmed-parents` be
> announced between Alice and Bob ?
>
> Anyway, I think it would be interesting to pseudo-specify the
> package-assemblage algorithm (or if there is code already available) to see
> if it's robust against adversarial or unlucky situations ?
>
> > In fact, a package
> > of transactions may be announced using both Erlay and package relay.
> > After reconciliation, if the initiator would have announced a
> > transaction by wtxid but also has package information for it, they may
> > send "inv(MSG_PCKG)" instead of "inv(WTX)".
>
> Yes, I think this holds. Note, we might have to add to the reconciliation
> set low-fee parents succeeding the feefilter check due to a child. When the
> reconcildiff, we might have to bifucarte again on feefilter to decide to
> announce missing wtixds either as `inv(MSG_PCKG)` or `inv(WTX)`.
>
> (IIRC, I've already made few feedbacks offline though good to get them in
> the public space and think more)
>
> Antoine
>
> Le mar. 17 mai 2022 à 12:09, Gloria Zhao via bitcoin-dev <
> bitcoin-dev@lists•linuxfoundation.org> a écrit :
>
>> Hi everybody,
>>
>> I’m writing to propose a set of p2p protocol changes to enable package
>> relay, soliciting feedback on the design and approach. Here is a link
>> to the most up-to-date proposal:
>>
>> https://github.com/bitcoin/bips/pull/1324
>>
>> If you have concept or approach feedback, *please respond on the
>> mailing list* to allow everybody to view and participate in the
>> discussion. If you find a typo or inaccurate wording, please feel free
>> to leave suggestions on the PR.
>>
>> I’m also working on an implementation for Bitcoin Core.
>>
>>
>> The rest of this post will include the same contents as the proposal,
>> with a bit of reordering and additional context. If you are not 100%
>> up-to-date on package relay and find the proposal hard to follow, I
>> hope you find this format more informative and persuasive.
>>
>>
>> ==Background and Motivation==
>>
>> Users may create and broadcast transactions that depend upon, i.e.
>> spend outputs of, unconfirmed transactions. A “package” is the
>> widely-used term for a group of transactions representable by a
>> connected Directed Acyclic Graph (where a directed edge exists between
>> a transaction that spends the output of another transaction).
>>
>> Incentive-compatible mempool and miner policies help create a fair,
>> fee-based market for block space. While miners maximize transaction
>> fees in order to earn higher block rewards, non-mining users
>> participating in transaction relay reap many benefits from employing
>> policies that result in a mempool with the same contents, including
>> faster compact block relay and more accurate fee estimation.
>> Additionally, users may take advantage of mempool and miner policy to
>> bump the priority of their transactions by attaching high-fee
>> descendants (Child Pays for Parent or CPFP).  Only considering
>> transactions one at a time for submission to the mempool creates a
>> limitation in the node's ability to determine which transactions have
>> the highest feerates, since it cannot take into account descendants
>> until all the transactions are in the mempool. Similarly, it cannot
>> use a transaction's descendants when considering which of two
>> conflicting transactions to keep (Replace by Fee or RBF).
>>
>> When a user's transaction does not meet a mempool's minimum feerate
>> and they cannot create a replacement transaction directly, their
>> transaction will simply be rejected by this mempool. They also cannot
>> attach a descendant to pay for replacing a conflicting transaction.
>> This limitation harms users' ability to fee-bump their transactions.
>> Further, it presents a security issue in contracting protocols which
>> rely on **presigned**, time-sensitive transactions to prevent cheating
>> (HTLC-Timeout in LN Penalty [1] [2] [3], Unvault Cancel in Revault
>> [4], Refund Transaction in Discreet Log Contracts [5], Updates in
>> eltoo [6]). In other words, a key security assumption of many
>> contracting protocols is that all parties can propagate and confirm
>> transactions in a timely manner.
>>
>> In the past few years, increasing attention [0][1][2][3][6] has been
>> brought to **pinning attacks**, a type of censorship in which the
>> attacker uses mempool policy restrictions to prevent a transaction
>> from being relayed or getting mined.  TLDR: revocation transactions
>> must meet a certain confirmation target to be effective, but their
>> feerates are negotiated well ahead of broadcast time. If the
>> forecasted feerate was too low and no fee-bumping options are
>> available, attackers can steal money from their counterparties. I walk
>> through a concrete example for stealing Lightning HTLC outputs at
>> ~23:58 in this talk [7][8].  Note that most attacks are only possible
>> when the market for blockspace at broadcast time  demands much higher
>> feerates than originally anticipated at signing time. Always
>> overestimating fees may sidestep this issue temporarily (while mempool
>> traffic is low and predictable), but this solution is not foolproof
>> and wastes users' money. The feerate market can change due to sudden
>> spikes in traffic (e.g. huge 12sat/vB dump a few days ago [9]) or
>> sustained, high volume of Bitcoin payments (e.g.  April 2021 and
>> December 2017).
>>
>> The best solution is to enable nodes to consider packages of
>> transactions as a unit, e.g. one or more low-fee parent transactions
>> with a high-fee child, instead of separately. A package-aware mempool
>> policy can help determine if it would actually be economically
>> rational to accept a transaction to the mempool if it doesn't meet fee
>> requirements individually. Network-wide adoption of these policies
>> would create a more purely-feerate-based market for block space and
>> allow contracting protocols to adjust fees (and therefore mining
>> priority) at broadcast time.  Some support for packages has existed in
>> Bitcoin Core for years. Since v0.13, Bitcoin Core has used ancestor
>> packages instead of individual transactions to evaluate the incentive
>> compatibility of transactions in the mempool [10] and select them for
>> inclusion in blocks [11].
>>
>> Package Relay, the concept of {announcing, requesting, downloading}
>> packages between nodes on the p2p network, has also been discussed for
>> many years. The earliest public mention I can find is from 2015 [12].
>> The two most common use cases for package relay are fee-bumping
>> otherwise-too-low-fee transactions and reducing the amount of orphans.
>> It seems uncontroversial to say that everybody desires package relay
>> conceptually, with varying degrees of urgency. Lots of work has been
>> done by others over the past few years, from which I've taken
>> inspiration from [13][14][15][16].
>>
>> My approach has been to split the project into two components: (1) Package
>> Mempool Accept, which includes validation logic and mempool policy.
>> (3) Package Relay, which includes the p2p protocol changes.
>>
>> Progress so far:
>> After discussions with various developers of contracting protocols
>> (with heavier emphasis towards LN), it was determined that a
>> package containing a child with all of its unconfirmed parents
>> (child-with-unconfirmed-parents or 1-child-multi-parent package) would
>> be sufficient for their use case, i.e. fee-bumping presigned
>> transactions. A child-with-unconfirmed-parents package has several
>> properties that make many things easier to reason about.
>>
>> A few months ago, I proposed a set of policies for safe package
>> validation and fee assessment for packages of this restricted
>> topology [17]. A series of PRs implementing this proposal have
>> been merged into Bitcoin Core [18].
>>
>> Theoretically, developing a safe and incentive-compatible package
>> mempool acceptance policy is sufficient to solve this issue. Nodes
>> could opportunistically accept packages (e.g. by trying combinations
>> of transactions rejected from their mempools), but this practice would
>> likely be inefficient at best and open new Denial of Service attacks
>> at worst. Additional p2p messages may enable nodes to request and
>> share package validation-related information with one another in a
>> more communication-efficient way.
>>
>> Given that only package RBF remains for package mempool accept, and we
>> can make progress on p2p and mempool in parallel, I think it’s
>> appropriate to put forward a package relay proposal.
>>
>> ==Proposal==
>>
>> This proposal contains 2 components: a “generic” package relay
>> protocol and an extension of it, child-with-unconfirmed-parents
>> packages, as version 1 package relay. Another version of packages,
>> “tx-with-unconfirmed-ancestors” can be created to extend package relay
>> for eliminating orphans.
>>
>> ===Generic Package Relay===
>>
>> Two main ideas are introduced:
>>
>> Download and validate packages of transactions together.
>>
>> Provide information to help peers decide whether to request and/or how
>> to validate transactions which are part of a package.
>>
>> ====Intended Protocol Flow====
>>
>> Due to the asynchronous nature of a distributed transaction relay
>> network, nodes may not receive all of the information needed to
>> validate a transaction at once. For example, after a node completes
>> Initial Block Download (IBD) and first starts participating in
>> transaction relay with an empty mempool, it is common to receive
>> orphans. In such scenarios where a node is aware that it is missing
>> information, a ''receiver-initiated'' dialogue is appropriate:
>>
>> 1. Receiver requests package information.
>>
>> 2. The sender provides package information, including the wtxids of
>>    the transactions in the package and anything else that might be
>> relevant (e.g. total fees and size).
>>
>> 3. The reciever uses the package information to decide how to request
>>    and validate the transactions.
>>
>> Sometimes, no matter what order transactions are received by a node,
>> validating them individually is insufficient. When the sender is aware
>> of additional information that the receiver needs to accept a package,
>> a proactive ''sender-initiated'' dialogue should be enabled:
>>
>> 1. Sender announces they have package information pertaining to a
>>    transaction that might otherwise be undesired on its own.
>>
>> 2. The receiver requests package information.
>>
>> 3. The sender provides package information, including the wtxids of
>>    the transactions in the package and anything else that might be
>> relevant (e.g. total fees and size).
>>
>> 4. The reciever uses the package information to decide how to request
>>    and validate the transactions.
>>
>> Package relay is negotiated between two peers during the version
>> handshake. Package relay requires both peers to support wtxid-based
>> relay because package transactions are referenced by their wtxid.
>>
>> ====New Messages====
>>
>> Three new protocol messages are added for use in any version of
>> package relay. Additionally, each version of package relay must define
>> its own inv type and "pckginfo" message version, referred to in this
>> document as "MSG_PCKG" and "pckginfo" respectively. See
>> BIP-v1-packages for a concrete example.
>>
>> =====sendpackages=====
>>
>> {|
>> |  Field Name  ||  Type  ||  Size  ||  Purpose
>> |-
>> |version || uint32_t || 4 || Denotes a package version supported by the
>> node.
>> |-
>> |max_count || uint32_t || 4 ||Specifies the maximum number of
>> transactions per package this node is
>> willing to accept.
>> |-
>> |max_weight || uint32_t || 4 ||Specifies the maximum total weight per
>> package this node is willing
>> to accept.
>> |-
>> |}
>>
>> 1. The "sendpackages" message has the structure defined above, with
>>    pchCommand == "sendpackages".
>>
>> 2. During version handshake, nodes should send a "sendpackages"
>>    message indicate they support package relay and may request
>> packages.
>>
>> 3. The message should contain a version supported by the node. Nodes
>>    should send a "sendpackages" message for each version they support.
>>
>> 4. The "sendpackages" message MUST be sent before sending a "verack"
>>    message. If a "sendpackages" message is received afer "verack", the
>> sender should be disconnected.
>>
>> 5. If 'fRelay==false' in a peer's version message, the node must not
>>    send "sendpackages" to them. If a "sendpackages" message is
>> received by a peer after sending `fRelay==false` in their version
>> message, the sender should be disconnected.
>>
>> 6.. Upon receipt of a "sendpackages" message with a version that is
>> not supported, a node must treat the peer as if it never received the
>> message.
>>
>> 7. If both peers send "wtxidrelay" and "sendpackages" with the same
>>    version, the peers should announce, request, and send package
>> information to each other.
>>
>> =====getpckgtxns=====
>>
>> {|
>> |  Field Name  ||  Type  ||  Size  ||   Purpose
>> |-
>> |txns_length||CompactSize||1 or 3 bytes|| The number of transactions
>> requested.
>> |-
>> |txns||List of wtxids||txns_length * 32|| The wtxids of each transaction
>> in the package.
>> |}
>>
>> 1. The "getpckgtxns" message has the structure defined above, with
>>    pchCommand == "getpckgtxns".
>>
>> 2. A "getpckgtxns" message should be used to request all or some of
>>    the transactions previously announced in a "pckginfo" message,
>> specified by witness transactiosome id.
>>
>> 3. Upon receipt of a "getpckgtxns" message, a node must respond with
>>    either a "pckgtxns" containing the requested transactions or a
>> "notfound" message indicating one or more of the transactions is
>> unavailable. This allows the receiver to avoid downloading and storing
>> transactions that cannot be validated immediately.
>>
>> 4. A "getpckgtxns" message should only be sent if both peers agreed to
>>    send packages in the version handshake. If a "getpckgtxns" message
>> is received from a peer with which package relay was not negotiated,
>> the sender should be disconnected.
>>
>> =====pckgtxns=====
>>
>> {|
>> |  Field Name  ||  Type  ||  Size  ||   Purpose
>> |-
>> |txns_length||CompactSize||1 or 3 bytes|| The number of transactions
>> provided.
>> |-
>> |txns||List of transactions||variable|| The transactions in the package.
>> |}
>>
>> 1. The "pckgtxns" message has the structure defined above, with
>>    pchCommand == "pckgtxns".
>>
>> 2. A "pckgtxns" message should contain the transaction data requested
>>    using "getpckgtxns".
>>
>> 3. A "pckgtxns" message should only be sent to a peer that requested
>>    the package using "getpckgtxns". If a node receives an unsolicited
>> package, the sender should be disconnected.
>>
>> 4. A "pckgtxns" message should only be sent if both peers agreed to
>>    send packages in the version handshake. If a "pckgtxns" message is
>> received from a peer with which package relay was not negotiated, the
>> sender should be disconnected.
>>
>> ===Version 1 Packages: child-with-unconfirmed-parents===
>>
>> This extends package relay for packages consisting of one transaction
>> and all of its unconfirmed parents,by defining version 1 packages, a
>> pckginfo1 message, and a MSG_PCKG1 inv type. It enables the use case
>> in which a child pays for its otherwise-too-low-fee parents and their
>> mempool conflict(s).
>>
>> ====Intended Protocol Flow====
>>
>> When relaying a package of low-fee parent(s) and high-fee child, the
>> sender and receiver do the following:
>>
>> 1. Sender announces they have a child-with-unconfirmed-parents package
>>    for a child that pays for otherwise-too-low-fee parent(s) using
>> "inv(MSG_PCKG1)".
>>
>> 2. The receiver requests package information using
>>    "getdata(MSG_PCKG1)".
>>
>> 3. The sender provides package information using "pckginfo1",
>>    including the blockhash of the sender's best block, the wtxids of
>> the transactions in the package, their total fees and total weight.
>>
>> 4. The reciever uses the package information to decide how to request
>>    the transactions. For example, if the receiver already has some of
>> the transactions in their mempool, they only request the missing ones.
>> They could also decide not to request the package at all based on the
>> fee information provided.
>>
>> 5. Upon receiving a "pckgtxns", the receiver submits the transactions
>>    together as a package.
>>
>> ====New Messages====
>>
>> A new inv type, "MSG_PCKG1", and new protocol message, "PCKGINFO1",
>> are added.
>>
>> =====pckginfo1=====
>>
>> {|
>> |  Field Name  ||  Type  ||  Size  ||   Purpose
>> |-
>> |blockhash || uint256 || 32 || The chain tip at which this package is
>> defined.
>> |-
>> |pckg_fee||CAmount||4|| The sum total fees paid by all transactions in
>> the package.
>> |-
>> |pckg_weight||int64_t||8|| The sum total weight of all transactions in
>> the package.
>> |-
>> |txns_length||CompactSize||1 or 3 bytes|| The number of transactions
>> provided.
>> |-
>> |txns||List of wtxids||txns_length * 32|| The wtxids of each transaction
>> in the package.
>> |}
>>
>>
>> 1. The "pckginfo1" message has the structure defined above, with
>>    pchCommand == "pckginfo1".
>>
>> 2. A "pckginfo1" message contains information about a version 1
>>    package (defined below), referenced by the wtxid of the transaction
>> it pertains to and the current blockhash.
>>
>> 3. Upon receipt of a "pckginfo1" message, the node should decide if it
>>    wants to validate the package, request transaction data if
>> necessary, etc.
>>
>> 4. Upon receipt of a malformed "pckginfo1" message or package that
>>    does not abide by the max_count, max_weight, or other rules
>> specified by the version agreed upon in the initial negotiation, the
>> sender should be disconnected.  If a node receives a "pckginfo1"
>> message for which the "pckg_fee" or "pckg_weight" do not reflect the
>> true total fees and weight, respectively, or the transactions in the
>> package, the message is malformed.
>>
>> 5. A node MUST NOT send a "pckginfo1" message that has not been
>>    requested by the recipient. Upon receipt of an unsolicited
>> "pckginfo1", a node should disconnect the sender.
>>
>> 6. A "pckginfo1" message should only be sent if both peers agreed to
>>    send version 1 packages in the version handshake. If a "pckginfo1"
>> message is received from a peer with which package relay was not
>> negotiated, the sender should be disconnected.
>>
>> =====MSG_PCKG1=====
>>
>> 1. A new inv type (MSG_PCKG1 == 0x6) is added, for use in inv messages
>>    and getdata requests pertaining to version 1 packages.
>>
>> 2. As an inv type, it indicates that both transaction data and version
>>    1 package information are available for the transaction. The
>> transaction is referenced by its wtxid. As a getdata request type, it
>> indicates that the sender wants package information for the
>> transaction.
>>
>> 3. Upon receipt of a "getdata" request for "MSG_PCKG1", the node
>>    should respond with the version 1 package corresponding to the
>> requested transaction and its current chain tip, or with NOTFOUND.
>> The node should not assume that the sender is requesting the
>> transaction data as well.
>>
>> ====Child With Parent Packages Rules====
>>
>> A child-with-unconfirmed-parents package sent between nodes must abide
>> by the rules below, otherwise the package is malformed and the sender
>> should be disconnected.
>>
>> A version 1 or ''child-with-unconfirmed-parents'' package can be
>> defined for any transaction that spends unconfirmed inputs. The child
>> can be thought of as the "representative" of the package. This package
>> can be uniquely identified by the transaction's wtxid and the current
>> chain tip block hash.
>>
>> A ''child-with-unconfirmed-parents'' package MUST be:
>>
>> 1. ''Sorted topologically.'' For every transaction t in the package,
>>    if any of t's parents are present in the package, the parent must
>> appear somewhere in the list before t. In other words, the
>> transactions must be sorted in ascending order of the number of
>> ancestors present in the package.
>>
>> 2. ''Only 1 child with unconfirmed parents.'' The package must consist
>>    of one transaction and its unconfirmed parents. There must not be
>> any other transactions in the package. Other dependency relationships
>> may exist within the package (e.g. one parent may spend the output of
>> another parent) provided that topological order is respected.
>>
>> 3. ''All unconfirmed parents.'' All of the child's unconfirmed parents
>>    must be present.
>>
>> 4. ''No conflicts.'' None of the transactions in the package may
>>    conflict with each other (i.e.  spend the same prevout).
>>
>> 5. ''Total fees and weight.'' The 'total_fee' and 'total_weight'
>>    fields must accurately represent the sum total of all transactions'
>> fees and weights as defined in BIP141, respectively.
>>
>> Not all of the child's parents must be present; the child transaction
>> may also spend confirmed inputs. However, if the child has confirmed
>> parents, they must not be in the package.
>>
>> While a child-with-unconfirmed-parents package is perhaps most
>> relevant when the child has a higher feerate than its parents, this
>> property is not required to construct a valid package.
>>
>> ====Clarifications====
>>
>> ''Q: Under what circumstances should a sender announce a
>> child-with-unconfirmed-parents package?''
>>
>> A child-with-unconfirmed-parents package for a transaction should be
>> announced when it meets the peer's fee filter but one or more of its
>> parents don't; a "inv(MSG_PCKG1)" instead of "inv(WTX)" should be sent
>> for the child. Each of the parents which meet the peer's fee filter
>> should still be announced normally.
>>
>> ''Q: What if a new block arrives in between messages?''
>>
>> A child-with-unconfirmed-parents package is defined for a transaction
>> based on the current chain state. As such, a new block extending the
>> tip may decrease the number of transactions in the package (i.e. if
>> any of the transaction's parents were included in the block). In a
>> reorg, the number of transactions in the package may decrease or
>> increase (i.e. if any of the transaction's parents were included in a
>> block in the previous chain but not the new one).
>>
>> If the new block arrives before the "getdata" or "pckginfo1", nothing
>> needs to change.
>>
>> If the new block arrives before "getpckgtxns" or before "pckgtxns",
>> the receiver may need to re-request package information if the block
>> contained a transaction in the package. If the block doesn't contain
>> any transactions in the package, whether it extends the previous tip
>> or causes a reorg, nothing needs to change.
>>
>> ''Q: Can "getpckgtxns" and "pckgtxns" messages contain only one
>> transaction?''
>>
>> Yes.
>>
>> ===Further Protocol Extensions===
>>
>> When introducing a new type of package, assign it a version number "n"
>> and use an additional "sendpackages" message during version handshake
>> to negotiate support for it. An additional package information message
>> "pckginfon" and inv type "MSG_PCKGn" should be defined for the type of
>> package.  However, "getpckgtxns" and "pckgtxns" do not need to be
>> changed.
>>
>> Example proposal for tx-with-unconfirmed-ancestors package relay: [19]
>>
>> ===Compatibility===
>>
>> Older clients remain fully compatible and interoperable after this
>> change. Clients implementing this protocol will only attempt to send
>> and request packages if agreed upon during the version handshake.
>>
>> ===Package Erlay===
>>
>> Clients using BIP330 reconciliation-based transaction relay (Erlay)
>> are able to use package relay without interference. In fact, a package
>> of transactions may be announced using both Erlay and package relay.
>> After reconciliation, if the initiator would have announced a
>> transaction by wtxid but also has package information for it, they may
>> send "inv(MSG_PCKG)" instead of "inv(WTX)".
>>
>> ===Rationale===
>>
>> ====P2P Message Design====
>>
>> These p2p messages are added for communication efficiency and, as
>> such, one should measure alternative solutions based on the resources
>> used to communicate (not necessarily trustworthy) information: We
>> would like to minimize network bandwidth, avoid downloading a
>> transaction more than once, avoid downloading transactions that are
>> eventually rejected, and minimize storage allocated for
>> not-yet-validated transactions.
>>
>> Consider these (plausible) scenarios in transaction relay:
>>
>> Alice (the "sender") is relaying transactions to Bob (the "receiver").
>> Alice's mempool has a minimum feerate of 1sat/vB and Bob's has a
>> minimum feerate of 3sat/vB. For simplicity, all transactions are
>> 1600Wu in virtual size and 500 bytes in serialized size. Apart from
>> the spending relationships specified, all other inputs are from
>> confirmed UTXOs.
>>
>> 1. Package {A, B} where A pays 0 satoshis and B pays 8000 satoshis in
>>    fees.
>>
>> 2. Package {C, D} where C pays 0 satoshis and D pays 1200 satoshis in
>>    fees.
>>
>> 3. Package {E, F, G, H, J} that pays 4000, 8000, 0, 2000, and 4000
>>    satoshis in fees, respectively.
>>
>> ====Alternative Designs Considered====
>>
>> ''Package Information Only:'' Just having "pckginfo" gives enough
>> information for the receiver to accept the package. Omit the
>> "getpckgtxns" and "pckgtxns" messages. While this option is a good
>> fallback if batched transaction download fails for some reason, it
>> shouldn't be used as the default because it 'always' requires storage
>> of unvalidated transactions.
>>
>> ''No Package Information Round:'' Instead of having a package
>> information round, just use the child's wtxid to refer to the package
>> and always send the entire package together. This would cause nodes to
>> redownload duplicate transactions.
>>
>> I have also created a slidedeck exploring various alternative designs
>> and some examples in which they fall flat [20]. Please feel free to
>> suggest other alternatives.
>>
>> ====Versioning System====
>>
>> This protocol should be extensible to support multiple types of
>> packages based on future desired use cases. Two "flavors" of
>> versioning were considered:
>>
>> 1. When package mempool acceptance is upgraded to support more types
>>    of packages, increment the version number (similar to Erlay).
>> During version handshake, peers negotiate which version of package
>> relay they will use by each sending one "sendpackages" message.
>>
>> 2. When introducing another type of package, assign a version number
>>    to it and announce it as an additional supported version (similar
>> to Compact Block Relay). During version handshake, peers send one
>> "sendpackages" message for each version supported.
>>
>> The second option was favored because it allows different parameters
>> for different versions.  For example, it should be possible to support
>> both "arbitrary topology but maximum 3-transaction" package as well as
>> "child-with-unconfirmed-parents with default mempool ancestor limits"
>> packages simultaneously.
>>
>> ==Acknowledgements==
>>
>> I hope to have made it abundantly clear that this proposal isn’t
>> inventing the concept of package relay, and in fact builds upon years
>> of work by many others, including Suhas Daftuar and Antoine Riard.
>>
>> Thank you to John Newbery and Martin Zumsande for input on the design.
>>
>> Thank you to Matt Corallo, Christian Decker, David Harding, Antoine
>> Poinsot, Antoine Riard, Gregory Sanders, Chris Stewart, Bastien
>> Teinturier, and others for input on the desired interface for
>> contracting protocols.
>>
>> Looking forward to hearing your thoughts!
>>
>> Best,
>> Gloria
>>
>> [0]:
>> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2022-January/019817.html
>> [1]:
>> https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-April/002639.html
>> [2]:
>> https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-June/002758.html
>> [3]:
>> https://github.com/t-bast/lightning-docs/blob/master/pinning-attacks.md
>> [4]:
>> https://github.com/revault/practical-revault/blob/master/transactions.md#cancel_tx
>> [5]:
>> https://github.com/discreetlogcontracts/dlcspecs/blob/master/Transactions.md#refund-transaction
>> [6]: https://gist.github.com/instagibbs/60264606e181451e977e439a49f69fe1
>> [7]:
>> https://btctranscripts.com/adopting-bitcoin/2021/2021-11-16-gloria-zhao-transaction-relay-policy/#lightning-attacks
>> [8]: https://youtu.be/fbWSQvJjKFs?t=1438
>> [9]:
>> https://www.reddit.com/r/Bitcoin/comments/unew4e/looks_like_70_mvb_of_transactions_just_got_dumped/
>> [10]: https://github.com/bitcoin/bitcoin/pull/7594
>> [11]: https://github.com/bitcoin/bitcoin/pull/7600
>> [12]: https://github.com/bitcoin/bitcoin/pull/6455#issuecomment-122716820
>> [13]: https://gist.github.com/sdaftuar/8756699bfcad4d3806ba9f3396d4e66a
>> [14]: https://github.com/bitcoin/bitcoin/issues/14895
>> [15]: https://github.com/bitcoin/bitcoin/pull/16401
>> [16]: https://github.com/bitcoin/bitcoin/pull/19621
>> [17]:
>> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2021-September/019464.html
>> [18]: https://github.com/users/glozow/projects/5/views/4?layout=board
>> [19]: https://gist.github.com/glozow/9b321cd3ef6505135c763112033ff2a7
>> [20]:
>> https://docs.google.com/presentation/d/1B__KlZO1VzxJGx-0DYChlWawaEmGJ9EGApEzrHqZpQc/edit?usp=sharing
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists•linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>

[-- Attachment #2: Type: text/html, Size: 45883 bytes --]

 permalink raw reply	[|nested]
2023-05-10 15:42 ` Greg Sanders
  
From: Tom Trevethan @ 2023-05-10 15:12 UTC (permalink / raw)
  To: Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 281 bytes --]

The submitpackage RPC is available on regtest in the current core release.
Is there any plan or timeline for deploying this on mainnet in the next
release? Can't find any recent discussion. It would be very helpful given
current (and likely future) issues with mempool purge.

Tom

[-- Attachment #2: Type: text/html, Size: 978 bytes --]

 permalink raw reply	[|nested]
2023-05-10 15:12 [bitcoin-dev] Package Relay Proposal Tom Trevethan

  
From: Greg Sanders @ 2023-05-10 15:42 UTC ( / )
  To: Tom Trevethan, Bitcoin Protocol Discussion



Hi Tom,

Yesterday a PR was opened to do just that, with caveats:
https://github.com/bitcoin/bitcoin/pull/27609

For higher level tracking of the project:
https://github.com/bitcoin/bitcoin/issues/27463

Cheers,
Greg

On Wed, May 10, 2023 at 11:39 AM Tom Trevethan via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> wrote:

> The submitpackage RPC is available on regtest in the current core
> release. Is there any plan or timeline for deploying this on mainnet in the
> next release? Can't find any recent discussion. It would be very helpful
> given current (and likely future) issues with mempool purge.
>
> Tom
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>