BIP-0322 (generic signmessage) improvements

BIP-0322 (generic signmessage) improvements
2020-12-21  5:37 ` Karl-Johan Alm
  
From: Andrew Poelstra @ 2020-12-18 15:27 UTC (permalink / raw)
  To: bitcoin-dev

[-- Attachment #1: Type: text/plain, Size: 13616 bytes --]

I have gone over BIP-0322 and substantially rewritten the text.
Everything I did is (I think) simply clarifying the existing
protocol, which felt like it was written by committee and wasn't
easy to follow, EXCEPT:

1. I rewrote the motivation section, which I believe originally
   was a paraphrase of Luke-jr's general objections to having any
   signmessage functionality. I hope Luke in particular can take
   a look at what I wrote under "Motivation" and see if it
   captures his concerns.

2. I merged the "consensus" and "upgradeable" rules to simply be
   one set of rules, consisting of consensus checks plus additional
   restrictions, all of which must be included. The new "Extensions"
   section allows validators to output the state "consensus-valid"
   if they really don't want to check the additional restrictions.

3. The "inconclusive" state, which was originally used for what I've
   called "consensus-valid", now indicates that a validator does not
   understand the script that it is checking (also described in the
   new "Extensions" section). The goal is that implementors are able
   to be meaningfully BIP-0322 while only supporting a subset of
   Script, e.g. the templates that their own software supports, or
   Miniscript, or the non-raw non-address set of output descriptors,
   or whatever.

   We have seen opposition to supporting BIP-322, e.g. [1] because
   of the requirement that you either have a full script interpreter
   (plus an open-ended list of Core's standardness flags, which is
   not even available through libbitcoinconsensus) or nothing. On
   the other hand, the vast majority of outputs are single-key p2pkh,
   p2pkwh or p2sh-wpkh.

The new text is here (and for posterity I will also include it
inline below, though unless Github deletes it it will be easier
to read in rendered form):

https://github.com/apoelstra/bips/blob/2020-12--bip322-overhaul/bip-0322.mediawiki

I'll also PR this to the BIPs repo in the next day or two, and
comments on Github are then welcome.


[1] https://bitcointalk.org/index.php?topic=5261605.0



* * * * * Full text of the above link * * * * *

<pre>
  BIP: 322
  Layer: Applications
  Title: Generic Signed Message Format
  Author: Karl-Johan Alm <karljohan-alm@garage•co.jp>
  Comments-Summary: No comments yet.
  Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-0322
  Status: Draft
  Type: Standards Track
  Created: 2018-09-10
  License: CC0-1.0
</pre>

== Abstract ==

A standard for interoperable signed messages based on the Bitcoin Script format, either for proving fund availability, or committing to a message as the intended recipient of funds sent to the invoice address.

== Motivation ==

The current message signing standard only works for P2PKH (1...) invoice addresses. We propose to extend and generalize the standard by using a Bitcoin Script based approach. This ensures that any coins, no matter what script they are controlled by, can in-principle be signed for. For easy interoperability with existing signing hardware, we also define a signature message format which resembles a Bitcoin transaction (except that it contains an invalid input, so it cannot be spent on any real network).

Additionally, the current message signature format uses ECDSA signatures which do not commit to the public key, meaning that they do not actually prove knowledge of any secret keys. (Indeed, valid signatures can be tweaked by 3rd parties to become valid signatures on certain related keys.)

Ultimately no message signing protocol can actually prove control of funds, both because a signature is obsolete as soon as it is created, and because the possessor of a secret key may be willing to sign messages on others' behalf even if it would not sign actual transactions. No signmessage protocol can fix these limitations.

== Types of Signatures ==

This BIP specifies three formats for signing messages: ''legacy'', ''simple'' and ''full''. Additionally, a variant of the ''full'' format can be used to demonstrate control over a set of UTXOs.

=== Legacy ===

New proofs should use the new format for all invoice address formats, including P2PKH.

The legacy format MAY be used, but must be restricted to the legacy P2PKH invoice address format.

=== Simple ===

A ''simple'' signature consists of a witness stack, consensus encoded as a vector of vectors of bytes, and base64-encoded. Validators should construct <code>to_spend</code> and <code>to_sign</code> as defined below, with default values for all fields except that

* <code>message_hash</code> is a BIP340-tagged hash of the message, as specified below
* <code>message_challenge</code> in <code>to_spend</code> is set to the scriptPubKey being signed with
* <code>message_signature</code> in <code>to_sign</code> is set to the provided simple signature.

and then proceed as they would for a full signature.

=== Full ===

Full signatures follow an analogous specification to the BIP-325 challenges and solutions used by Signet.

Let there be two virtual transactions to_spend and to_sign.

The "to_spend" transaction is:

    nVersion = 0
    nLockTime = 0
    vin[0].prevout.hash = 0000...000
    vin[0].prevout.n = 0xFFFFFFFF
    vin[0].nSequence = 0
    vin[0].scriptSig = OP_0 PUSH32[ message_hash ]
    vin[0].scriptWitness = []
    vout[0].nValue = 0
    vout[0].scriptPubKey = message_challenge

where <code>message_hash</code> is a BIP340-tagged hash of the message, i.e. sha256_tag(m), where tag = <code>BIP0322-signed-message</code>, and <code>message_challenge</code> is the to be proven (public) key script.

The "to_sign" transaction is:

    nVersion = 0 or as appropriate (e.g. 2, for time locks)
    nLockTime = 0 or as appropriate (for time locks)
    vin[0].prevout.hash = to_spend.txid
    vin[0].prevout.n = 0
    vin[0].nSequence = 0 or as appropriate (for time locks)
    vin[0].scriptWitness = message_signature
    vout[0].nValue = 0
    vout[0].scriptPubKey = OP_RETURN

A full signature consists of the base64-encoding of the to_spend and to_sign transactions concatenated in standard network serialisation.

=== Full (Proof of Funds) ===

A signer may construct a proof of funds, demonstrating control of a set of UTXOs, by constructing a full signature as above, with the following modifications.

* <code>message_challenge</code> is unused and shall be set to <code>OP_TRUE</code>
* Similarly, <code>message_signature</code> is then empty.
* All outputs that the signer wishes to demonstrate control of are included as additional outputs to <code>to_sign</code>, and their witness and scriptSig data should be set as though these outputs were actually being spent.

Unlike an ordinary signature, validators of a proof of funds need access to the current UTXO set, to learn that the claimed inputs exist on the blockchain, and to learn their scriptPubKeys.

== Detailed Specification ==

For all signature types, except legacy, the <code>to_spend</code> and <code>to_sign</code> transactions must be valid transactions which pass all consensus checks, except of course that the output with prevout <code>000...000:FFFFFFFF</code> does not exist.

We additionally require the following restrictions be met.

* All signatures must use the SIGHASH_ALL flag.
* The use of <code>CODESEPARATOR</code> or <code>FindAndDelete</code> is forbidden.
* The use of NOPs reserved for upgrades is forbidden.
* The use of segwit versions greater than 1 are forbidden.
* <code>LOW_S</code>, <code>STRICTENC</code> and <code>NULLFAIL</code>: valid ECDSA signatures must be strictly DER-encoded and have a low-S value; invalid ECDSA signature must be the empty push
* <code>MINIMALDATA</code>: all pushes must be minimally encoded
* <code>CLEANSTACK</code>: require that only a single stack element remains after evaluation
* <code>MINIMALIF</code>: the argument of <code>IF</code>/<code>NOTIF</code> must be exactly 0x01 or empty push

Future versions of this BIP may relax these rules, in particular those around NOPs and future Segwit versions, as they are deployed on Bitcoin.

=== Verification ===

Validation consists of the following steps. A validator is given as input an address ''A'' (which may be omitted in a proof-of-funds), signature ''s'' and message ''m'', and outputs one of four states (although validators are only required to be able to output the first and last):
* ''valid'' indicates that the signature passed all checks described below
* ''valid at time t and age s'' indicates that the signature has set timelocks but is otherwise valid (see "Extensions" below)
* ''consensus-valid'' indicates that the signature passed validation except for the additonal restrictions in the above section (see "Extensions" below)
* ''inconclusive'' means the validator was unable to check the scripts (see "Extensions" below)
* ''invalid'' means none of the other states

# Decode ''s'' as the transactions <code>to_sign</code> and <code>to_spend</code>
# Confirm that <code>message_hash</code> is the correct hash of ''m''
# Confirm that <code>message_challenge</code> is the scriptPubKey corresponding to ''A'' if ''A'' is present, and otherwise must be <code>OP_TRUE</code>
# Confirm that all other fields are set as specified above; in particular that
** <code>to_spend</code> has exactly one input and one output
** <code>to_sign</code> has at least one input and its first input spends the output of </code>to_spend</code>
** <code>to_sign</code> has exactly one output, as specified above
# Confirm that the two transactions together satisfy all consensus rules, except for <code>to_spend</code>'s missing input, and except that ''nSequence'' of <code>to_sign</code>'s first input and ''nLockTime'' of <code>to_sign</code> are not checked.
# Confirm that all of the above restrictions are met.

If the above conditions are met, the signature is considered ''valid''. Otherwise the signature is ''invalid''.

=== Signing ===

Signers who control an address ''A'' who wish to sign a message ''m'' act as follows:

# They construct <code>to_spend</code> and <code>to_sign</code> as specified above, using the scriptPubKey of ''A'' for <code>message_challenge</code> and tagged hash of ''m'' as <code>message_hash</code>.
# Optionally, they may set nLockTime of <code>to_sign</code> or nSequence of its first input.
# Optionally, they may add any additional outputs to <code>to_sign</code> that they wish to prove control of.
# They satisfy <code>to_sign</code> as they would any other transaction.

They then encode their signature, choosing either ''simple'' or ''full'' as follows:

* If they added no inputs to <code>to_sign</code>, left nSequence and nLockTime at 0, and ''A'' is a Segwit address (either pure or P2SH-wrapped), then they may base64-encode <code>message_signature</code>
* Otherwise they must base64-encode the concatenation of <code>to_spend</code> followed by <code>to_sign</code>.

== Extensions ==

To ease implementation, we allow some additional states to be output rather than ''valid'' or ''invalid''. Users who do not understand or who do not wish to deal with these states may treat them as ''invalid''.

=== Timelocks === 

If the nLockTime of <code>to_sign</code> is set to ''t'', and the nSequence of the first input of <code>to_sign</code> is set to ''s'', the validator may output the state ''valid at time t and age s''.

If both ''t'' and ''s'' are 0, the validator must instead output ''valid''.

Users may then wish to interpret this state as ''valid'' or ''invalid'' relative to the state of the current blockchain, but the rules for doing so are out of scope of this BIP.

=== Incomplete Validation ===

Some validators may not wish to implement a full script interpreter, choosing instead to support only specific script templates, or only Miniscript, for example. In this case, if they are unable to execute the scripts used by <code>to_sign</code>, they should output the state ''inconclusive''.

Users should interpret this state as the same thing as ''invalid'', although take it as a sign that they should find more capable software.

=== Consensus-Only Validation ===

Validators which are only able to check consensus-correctness of witnesses, but not the additional restrictions imposed by this BIP, may output the state ''consensus-valid'' to indicate that a signature has passed all consensus and structural checks.

Users should interpret this state as the same thing as ''valid'' but be aware that other software may fail to validate the same signature.

== Compatibility ==

This specification is backwards compatible with the legacy signmessage/verifymessage specification through the special case as described above.

== Reference implementation ==

TODO

== Acknowledgements ==

Thanks to David Harding, Jim Posen, Kalle Rosenbaum, Pieter Wuille, Andrew Poelstra, and many others for their feedback on the specification.

== References ==

# Original mailing list thread: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-March/015818.html

== Copyright ==

This document is licensed under the Creative Commons CC0 1.0 Universal license.

== Test vectors ==

TODO

* * * * * End full text * * * * *

-- 
Andrew Poelstra
Director of Research, Blockstream
Email: apoelstra at wpsoftware.net
Web:   https://www.wpsoftware.net/andrew

The sun is always shining in space
    -Justin Lewis-Webster


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

 permalink raw reply	[|nested]
2020-12-18 15:27 [bitcoin-dev] BIP-0322 (generic signmessage) improvements Andrew Poelstra

  2020-12-21 22:57   ` Pieter Wuille
  
From: Karl-Johan Alm @ 2020-12-21  5:37 UTC ( / )
  To: Andrew Poelstra, Bitcoin Protocol Discussion

Thanks a lot for taking the time to brush up the BIP. For what it's
worth, I am all for these changes, and I see them as clear
improvements all around.

IIRC Pieter was the one who originally suggested the two-checks
approach (invalid, inconclusive, valid) which is being modified here,
so would be good if you chimed in (or not -- which I'll assume means
you don't mind).

On Sat, Dec 19, 2020 at 12:27 AM Andrew Poelstra via bitcoin-dev
<bitcoin-dev@lists•linuxfoundation.org> wrote:
>
> I have gone over BIP-0322 and substantially rewritten the text.
> Everything I did is (I think) simply clarifying the existing
> protocol, which felt like it was written by committee and wasn't
> easy to follow, EXCEPT:
>
> 1. I rewrote the motivation section, which I believe originally
>    was a paraphrase of Luke-jr's general objections to having any
>    signmessage functionality. I hope Luke in particular can take
>    a look at what I wrote under "Motivation" and see if it
>    captures his concerns.
>
> 2. I merged the "consensus" and "upgradeable" rules to simply be
>    one set of rules, consisting of consensus checks plus additional
>    restrictions, all of which must be included. The new "Extensions"
>    section allows validators to output the state "consensus-valid"
>    if they really don't want to check the additional restrictions.
>
> 3. The "inconclusive" state, which was originally used for what I've
>    called "consensus-valid", now indicates that a validator does not
>    understand the script that it is checking (also described in the
>    new "Extensions" section). The goal is that implementors are able
>    to be meaningfully BIP-0322 while only supporting a subset of
>    Script, e.g. the templates that their own software supports, or
>    Miniscript, or the non-raw non-address set of output descriptors,
>    or whatever.
>
>    We have seen opposition to supporting BIP-322, e.g. [1] because
>    of the requirement that you either have a full script interpreter
>    (plus an open-ended list of Core's standardness flags, which is
>    not even available through libbitcoinconsensus) or nothing. On
>    the other hand, the vast majority of outputs are single-key p2pkh,
>    p2pkwh or p2sh-wpkh.
>
> The new text is here (and for posterity I will also include it
> inline below, though unless Github deletes it it will be easier
> to read in rendered form):
>
> https://github.com/apoelstra/bips/blob/2020-12--bip322-overhaul/bip-0322.mediawiki
>
> I'll also PR this to the BIPs repo in the next day or two, and
> comments on Github are then welcome.
>
>
> [1] https://bitcointalk.org/index.php?topic=5261605.0
>
>
>
> * * * * * Full text of the above link * * * * *
>
> <pre>
>   BIP: 322
>   Layer: Applications
>   Title: Generic Signed Message Format
>   Author: Karl-Johan Alm <karljohan-alm@garage•co.jp>
>   Comments-Summary: No comments yet.
>   Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-0322
>   Status: Draft
>   Type: Standards Track
>   Created: 2018-09-10
>   License: CC0-1.0
> </pre>
>
> == Abstract ==
>
> A standard for interoperable signed messages based on the Bitcoin Script format, either for proving fund availability, or committing to a message as the intended recipient of funds sent to the invoice address.
>
> == Motivation ==
>
> The current message signing standard only works for P2PKH (1...) invoice addresses. We propose to extend and generalize the standard by using a Bitcoin Script based approach. This ensures that any coins, no matter what script they are controlled by, can in-principle be signed for. For easy interoperability with existing signing hardware, we also define a signature message format which resembles a Bitcoin transaction (except that it contains an invalid input, so it cannot be spent on any real network).
>
> Additionally, the current message signature format uses ECDSA signatures which do not commit to the public key, meaning that they do not actually prove knowledge of any secret keys. (Indeed, valid signatures can be tweaked by 3rd parties to become valid signatures on certain related keys.)
>
> Ultimately no message signing protocol can actually prove control of funds, both because a signature is obsolete as soon as it is created, and because the possessor of a secret key may be willing to sign messages on others' behalf even if it would not sign actual transactions. No signmessage protocol can fix these limitations.
>
> == Types of Signatures ==
>
> This BIP specifies three formats for signing messages: ''legacy'', ''simple'' and ''full''. Additionally, a variant of the ''full'' format can be used to demonstrate control over a set of UTXOs.
>
> === Legacy ===
>
> New proofs should use the new format for all invoice address formats, including P2PKH.
>
> The legacy format MAY be used, but must be restricted to the legacy P2PKH invoice address format.
>
> === Simple ===
>
> A ''simple'' signature consists of a witness stack, consensus encoded as a vector of vectors of bytes, and base64-encoded. Validators should construct <code>to_spend</code> and <code>to_sign</code> as defined below, with default values for all fields except that
>
> * <code>message_hash</code> is a BIP340-tagged hash of the message, as specified below
> * <code>message_challenge</code> in <code>to_spend</code> is set to the scriptPubKey being signed with
> * <code>message_signature</code> in <code>to_sign</code> is set to the provided simple signature.
>
> and then proceed as they would for a full signature.
>
> === Full ===
>
> Full signatures follow an analogous specification to the BIP-325 challenges and solutions used by Signet.
>
> Let there be two virtual transactions to_spend and to_sign.
>
> The "to_spend" transaction is:
>
>     nVersion = 0
>     nLockTime = 0
>     vin[0].prevout.hash = 0000...000
>     vin[0].prevout.n = 0xFFFFFFFF
>     vin[0].nSequence = 0
>     vin[0].scriptSig = OP_0 PUSH32[ message_hash ]
>     vin[0].scriptWitness = []
>     vout[0].nValue = 0
>     vout[0].scriptPubKey = message_challenge
>
> where <code>message_hash</code> is a BIP340-tagged hash of the message, i.e. sha256_tag(m), where tag = <code>BIP0322-signed-message</code>, and <code>message_challenge</code> is the to be proven (public) key script.
>
> The "to_sign" transaction is:
>
>     nVersion = 0 or as appropriate (e.g. 2, for time locks)
>     nLockTime = 0 or as appropriate (for time locks)
>     vin[0].prevout.hash = to_spend.txid
>     vin[0].prevout.n = 0
>     vin[0].nSequence = 0 or as appropriate (for time locks)
>     vin[0].scriptWitness = message_signature
>     vout[0].nValue = 0
>     vout[0].scriptPubKey = OP_RETURN
>
> A full signature consists of the base64-encoding of the to_spend and to_sign transactions concatenated in standard network serialisation.
>
> === Full (Proof of Funds) ===
>
> A signer may construct a proof of funds, demonstrating control of a set of UTXOs, by constructing a full signature as above, with the following modifications.
>
> * <code>message_challenge</code> is unused and shall be set to <code>OP_TRUE</code>
> * Similarly, <code>message_signature</code> is then empty.
> * All outputs that the signer wishes to demonstrate control of are included as additional outputs to <code>to_sign</code>, and their witness and scriptSig data should be set as though these outputs were actually being spent.
>
> Unlike an ordinary signature, validators of a proof of funds need access to the current UTXO set, to learn that the claimed inputs exist on the blockchain, and to learn their scriptPubKeys.
>
> == Detailed Specification ==
>
> For all signature types, except legacy, the <code>to_spend</code> and <code>to_sign</code> transactions must be valid transactions which pass all consensus checks, except of course that the output with prevout <code>000...000:FFFFFFFF</code> does not exist.
>
> We additionally require the following restrictions be met.
>
> * All signatures must use the SIGHASH_ALL flag.
> * The use of <code>CODESEPARATOR</code> or <code>FindAndDelete</code> is forbidden.
> * The use of NOPs reserved for upgrades is forbidden.
> * The use of segwit versions greater than 1 are forbidden.
> * <code>LOW_S</code>, <code>STRICTENC</code> and <code>NULLFAIL</code>: valid ECDSA signatures must be strictly DER-encoded and have a low-S value; invalid ECDSA signature must be the empty push
> * <code>MINIMALDATA</code>: all pushes must be minimally encoded
> * <code>CLEANSTACK</code>: require that only a single stack element remains after evaluation
> * <code>MINIMALIF</code>: the argument of <code>IF</code>/<code>NOTIF</code> must be exactly 0x01 or empty push
>
> Future versions of this BIP may relax these rules, in particular those around NOPs and future Segwit versions, as they are deployed on Bitcoin.
>
> === Verification ===
>
> Validation consists of the following steps. A validator is given as input an address ''A'' (which may be omitted in a proof-of-funds), signature ''s'' and message ''m'', and outputs one of four states (although validators are only required to be able to output the first and last):
> * ''valid'' indicates that the signature passed all checks described below
> * ''valid at time t and age s'' indicates that the signature has set timelocks but is otherwise valid (see "Extensions" below)
> * ''consensus-valid'' indicates that the signature passed validation except for the additonal restrictions in the above section (see "Extensions" below)
> * ''inconclusive'' means the validator was unable to check the scripts (see "Extensions" below)
> * ''invalid'' means none of the other states
>
> # Decode ''s'' as the transactions <code>to_sign</code> and <code>to_spend</code>
> # Confirm that <code>message_hash</code> is the correct hash of ''m''
> # Confirm that <code>message_challenge</code> is the scriptPubKey corresponding to ''A'' if ''A'' is present, and otherwise must be <code>OP_TRUE</code>
> # Confirm that all other fields are set as specified above; in particular that
> ** <code>to_spend</code> has exactly one input and one output
> ** <code>to_sign</code> has at least one input and its first input spends the output of </code>to_spend</code>
> ** <code>to_sign</code> has exactly one output, as specified above
> # Confirm that the two transactions together satisfy all consensus rules, except for <code>to_spend</code>'s missing input, and except that ''nSequence'' of <code>to_sign</code>'s first input and ''nLockTime'' of <code>to_sign</code> are not checked.
> # Confirm that all of the above restrictions are met.
>
> If the above conditions are met, the signature is considered ''valid''. Otherwise the signature is ''invalid''.
>
> === Signing ===
>
> Signers who control an address ''A'' who wish to sign a message ''m'' act as follows:
>
> # They construct <code>to_spend</code> and <code>to_sign</code> as specified above, using the scriptPubKey of ''A'' for <code>message_challenge</code> and tagged hash of ''m'' as <code>message_hash</code>.
> # Optionally, they may set nLockTime of <code>to_sign</code> or nSequence of its first input.
> # Optionally, they may add any additional outputs to <code>to_sign</code> that they wish to prove control of.
> # They satisfy <code>to_sign</code> as they would any other transaction.
>
> They then encode their signature, choosing either ''simple'' or ''full'' as follows:
>
> * If they added no inputs to <code>to_sign</code>, left nSequence and nLockTime at 0, and ''A'' is a Segwit address (either pure or P2SH-wrapped), then they may base64-encode <code>message_signature</code>
> * Otherwise they must base64-encode the concatenation of <code>to_spend</code> followed by <code>to_sign</code>.
>
> == Extensions ==
>
> To ease implementation, we allow some additional states to be output rather than ''valid'' or ''invalid''. Users who do not understand or who do not wish to deal with these states may treat them as ''invalid''.
>
> === Timelocks ===
>
> If the nLockTime of <code>to_sign</code> is set to ''t'', and the nSequence of the first input of <code>to_sign</code> is set to ''s'', the validator may output the state ''valid at time t and age s''.
>
> If both ''t'' and ''s'' are 0, the validator must instead output ''valid''.
>
> Users may then wish to interpret this state as ''valid'' or ''invalid'' relative to the state of the current blockchain, but the rules for doing so are out of scope of this BIP.
>
> === Incomplete Validation ===
>
> Some validators may not wish to implement a full script interpreter, choosing instead to support only specific script templates, or only Miniscript, for example. In this case, if they are unable to execute the scripts used by <code>to_sign</code>, they should output the state ''inconclusive''.
>
> Users should interpret this state as the same thing as ''invalid'', although take it as a sign that they should find more capable software.
>
> === Consensus-Only Validation ===
>
> Validators which are only able to check consensus-correctness of witnesses, but not the additional restrictions imposed by this BIP, may output the state ''consensus-valid'' to indicate that a signature has passed all consensus and structural checks.
>
> Users should interpret this state as the same thing as ''valid'' but be aware that other software may fail to validate the same signature.
>
> == Compatibility ==
>
> This specification is backwards compatible with the legacy signmessage/verifymessage specification through the special case as described above.
>
> == Reference implementation ==
>
> TODO
>
> == Acknowledgements ==
>
> Thanks to David Harding, Jim Posen, Kalle Rosenbaum, Pieter Wuille, Andrew Poelstra, and many others for their feedback on the specification.
>
> == References ==
>
> # Original mailing list thread: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-March/015818.html
>
> == Copyright ==
>
> This document is licensed under the Creative Commons CC0 1.0 Universal license.
>
> == Test vectors ==
>
> TODO
>
> * * * * * End full text * * * * *
>
> --
> Andrew Poelstra
> Director of Research, Blockstream
> Email: apoelstra at wpsoftware.net
> Web:   https://www.wpsoftware.net/andrew
>
> The sun is always shining in space
>     -Justin Lewis-Webster
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
2020-12-21  5:37 ` Karl-Johan Alm

  2020-12-22  0:22     ` Pieter Wuille
  
From: Pieter Wuille @ 2020-12-21 22:57 UTC (permalink / raw)
  To: Karl-Johan Alm, Bitcoin Protocol Discussion

On Sunday, December 20, 2020 9:37 PM, Karl-Johan Alm via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:

> Thanks a lot for taking the time to brush up the BIP. For what it's
> worth, I am all for these changes, and I see them as clear
> improvements all around.
>
> IIRC Pieter was the one who originally suggested the two-checks
> approach (invalid, inconclusive, valid) which is being modified here,
> so would be good if you chimed in (or not -- which I'll assume means
> you don't mind).

I agree with the idea of permitting incomplete validators to return inconclusive as well. That doesn't really reduce the functionality (given that "inconclusive" was already a potential result), and it obviously makes it much more accessible to a variety of software.

This suggestion breaks the original use of inconclusive though: the ability to detect that future features are used in the signature. The idea was to use divergence between "consensus valid" and "standardness valid" as a proxy for future extensions to be detected (e.g. OP_NOPn, future witness versions, ...). I think it's undesirable that these things now become unconditionally invalid (until the BIP is updated, but once that happens old validators will give a different result than new ones).

Since the BIP no longer relies on a nebulous concept of standardness, and instead specifically defines which standardness features are to be considered, this seems easy to fix: whenever validation fails due to any of those, require reporting inconclusive instead of invalid (unless of course something actually invalid also happens). In practice I guess you'd implement that (in capable validators) by still doing validation twice, once with all features enabled that distinguish between valid/invalid, and if valid, again but now with the features enabled that distinguish between valid and (invalid or inconclusive).

Cheers,

--
Pieter



 permalink raw reply	[|nested]
2020-12-21 22:57   ` Pieter Wuille

  2020-12-22  1:11       ` Andrew Poelstra
  2020-12-23 15:55       ` Andrew Poelstra
  
From: Pieter Wuille @ 2020-12-22  0:22 UTC (permalink / raw)
  To: Pieter Wuille, Bitcoin Protocol Discussion

On Monday, December 21, 2020 2:57 PM, Pieter Wuille via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org> wrote:

> On Sunday, December 20, 2020 9:37 PM, Karl-Johan Alm via bitcoin-dev bitcoin-dev@lists•linuxfoundation.org wrote:
>
> > Thanks a lot for taking the time to brush up the BIP. For what it's
> > worth, I am all for these changes, and I see them as clear
> > improvements all around.
> > IIRC Pieter was the one who originally suggested the two-checks
> > approach (invalid, inconclusive, valid) which is being modified here,
> > so would be good if you chimed in (or not -- which I'll assume means
> > you don't mind).
>
> I agree with the idea of permitting incomplete validators to return inconclusive as well. That doesn't really reduce the functionality (given that "inconclusive" was already a potential result), and it obviously makes it much more accessible to a variety of software.
>
> This suggestion breaks the original use of inconclusive though: the ability to detect that future features are used in the signature. The idea was to use divergence between "consensus valid" and "standardness valid" as a proxy for future extensions to be detected (e.g. OP_NOPn, future witness versions, ...). I think it's undesirable that these things now become unconditionally invalid (until the BIP is updated, but once that happens old validators will give a different result than new ones).
>
> Since the BIP no longer relies on a nebulous concept of standardness, and instead specifically defines which standardness features are to be considered, this seems easy to fix: whenever validation fails due to any of those, require reporting inconclusive instead of invalid (unless of course something actually invalid also happens). In practice I guess you'd implement that (in capable validators) by still doing validation twice, once with all features enabled that distinguish between valid/invalid, and if valid, again but now with the features enabled that distinguish between valid and (invalid or inconclusive).

Re-reading your proposed text, I'm wondering if the "consensus-only validation" extension is intended to replace the inconclusive-due-to-consensus-and-standardness-differ state. If so, I don't think it does, and regardless it doesn't seem very useful.

What I'm suggestion could be specified this way:
* If validator understands the script:
  * If signature is consensus valid (as far as the validator knows):
    * If signature is not known to trigger standardness rules intended for future extension (well-defined set of rules listed in BIP, and revisable): return valid
    * Otherwise: return inconclusive
  * Otherwise: return invalid
* Otherwise: return inconclusive

Or in other words: every signature has a well-defined result (valid, invalid, inconclusive) + validators may choose to report inconclusive for anything they don't understand.

This has the property that as long as new consensus rules only change things that were covered under for-future-extension standardness rules, no two validators will ever claim valid and invalid for the same signature. Only valid+inconclusive or invalid+inconclusive.

Cheers,

--
Pieter



 permalink raw reply	[|nested]
2020-12-22  0:22     ` Pieter Wuille

  2020-12-23 15:55       ` Andrew Poelstra
  
From: Andrew Poelstra @ 2020-12-22  1:11 UTC (permalink / raw)
  To: Pieter Wuille, Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 2803 bytes --]

On Tue, Dec 22, 2020 at 12:22:37AM +0000, Pieter Wuille via bitcoin-dev wrote:
> 
> Re-reading your proposed text, I'm wondering if the "consensus-only validation" extension is intended to replace the inconclusive-due-to-consensus-and-standardness-differ state. If so, I don't think it does, and regardless it doesn't seem very useful.
> 
> What I'm suggestion could be specified this way:
> * If validator understands the script:
>   * If signature is consensus valid (as far as the validator knows):
>     * If signature is not known to trigger standardness rules intended for future extension (well-defined set of rules listed in BIP, and revisable): return valid
>     * Otherwise: return inconclusive
>   * Otherwise: return invalid
> * Otherwise: return inconclusive
> 
> Or in other words: every signature has a well-defined result (valid, invalid, inconclusive) + validators may choose to report inconclusive for anything they don't understand.
> 
> This has the property that as long as new consensus rules only change things that were covered under for-future-extension standardness rules, no two validators will ever claim valid and invalid for the same signature. Only valid+inconclusive or invalid+inconclusive.
>

I like it!

My thinking regarding standardness vs consensus rules was essentially that
I wanted to enforce the included standardness rules for anti-malleability
reasons, i.e. the hope that for "normal scripts" we would get strong signatures,
which may be important for anti-DoS reasons. (What I mean by this is that
if you can easily create mutations of signatures, it may confuse software
in similar ways to the Gox-era malleability attacks on wallet software of
the time.) But conversely, it is hard to enforce these rules as an
implementor, because libbitcoinconsensus does not expose them. So allowing
both forms of validation, to me, was an attempt to encourage adoption
rather than anything principled.

I didn't even consider the idea that validators should be able to signal "this
signature appears to use future consensus rules", although I should have been
clued in by your "upgradeable rules" language that this was your goal. Now that
you say this, it's obvious that this is desireable, and also obvious that using
the "inconclusive" state is an elegant way to achieve this.

I also agree that "confirming validators should never disagree on valid vs
invalid" is a good design goal and we should make that explicit.


I'll add a commit to my PR at https://github.com/bitcoin/bips/pull/1048 which
adds these thoughts.

-- 
Andrew Poelstra
Director of Research, Blockstream
Email: apoelstra at wpsoftware.net
Web:   https://www.wpsoftware.net/andrew

The sun is always shining in space
    -Justin Lewis-Webster


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

 permalink raw reply	[|nested]
2020-12-22  0:22     ` Pieter Wuille
  2020-12-22  1:11       ` Andrew Poelstra

  
From: Andrew Poelstra @ 2020-12-23 15:55 UTC (permalink / raw)
  To: Pieter Wuille, Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 2264 bytes --]

On Tue, Dec 22, 2020 at 12:22:37AM +0000, Pieter Wuille via bitcoin-dev wrote:
> 
> Re-reading your proposed text, I'm wondering if the "consensus-only validation" extension is intended to replace the inconclusive-due-to-consensus-and-standardness-differ state. If so, I don't think it does, and regardless it doesn't seem very useful.
> 
> What I'm suggestion could be specified this way:
> * If validator understands the script:
>   * If signature is consensus valid (as far as the validator knows):
>     * If signature is not known to trigger standardness rules intended for future extension (well-defined set of rules listed in BIP, and revisable): return valid
>     * Otherwise: return inconclusive
>   * Otherwise: return invalid
> * Otherwise: return inconclusive
> 
> Or in other words: every signature has a well-defined result (valid, invalid, inconclusive) + validators may choose to report inconclusive for anything they don't understand.
> 
> This has the property that as long as new consensus rules only change things that were covered under for-future-extension standardness rules, no two validators will ever claim valid and invalid for the same signature. Only valid+inconclusive or invalid+inconclusive.
>

I've updated my PR at https://github.com/bitcoin/bips/pull/1048

Differences:

1. I compacted all the validation states into three: valid at time/age T/S, invalid,
   and inconclusive.

2. "Inconclusive" means either an "upgradeable rule" failed, e.g. use of a NOP or a
   bad network version, or the validator just didn't understand the scripts.

3. I removed the "Extensions" sections now everything is in the main protocol.

4. I removed the "to_sign" transaction from the wire serialization, since after all
   this, it can always be inferred from the message and address. (This does mean,
   however, that there is no way to sign for scriptPubKeys that don't have addresses,
   e.g. bare public keys or multisigs. I don't think it's worth complicated the
   protocol for such obscure things.)

-- 
Andrew Poelstra
Director of Research, Blockstream
Email: apoelstra at wpsoftware.net
Web:   https://www.wpsoftware.net/andrew

The sun is always shining in space
    -Justin Lewis-Webster


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

 permalink raw reply	[|nested]