Re: Proposed Extensions to BIP 174 for Future Extensibility

Re: Proposed Extensions to BIP 174 for Future Extensibility
2019-07-31 14:32 ` jan matejek
  2019-07-31 16:19 ` Dmitry Petukhov
  
From: Andrew Chow @ 2019-07-31  1:13 UTC (permalink / raw)
  To: Bitcoin Protocol Discussion

Hi All,

I would like to propose some types that allow for BIP 174 PSBT to be
extended more in the future.


Firstly, I would like to propose that some types be reserved for
proprietary use. These proprietary use types are, in general, for
private use by individuals and organizations who want to use PSBT in
their processes. These are usefule when there are additional data that
they need attached to a PSBT but such data are not useful (or available)
for the general public.

These types will be guaranteed to not be used by the public
specification and there is no expectation that any publicly available
software be able to understand any specific meanings of these types.
These types should be used for internal processes only.

The types I would like to reserve for proprietary use are the 15 types
from 0xF0 to 0xFE inclusive. These 15 type values will be the same for
global, per-input, and per-output types. If 15 types are not enough,
additional types can be obtained using the multi-byte type method
described later.


Next, I would like to propose a global version type and field. The
version type is 0xEF with only the type as the key, and the value is a
32-bit unsigned little endian integer representing the version number. A
PSBT without a version number is to be considered version 0. If a parser
sees a version number that it does not understand, it should exit
immediately as the PSBT will contain types that are not safe to ignore.

This version number is a safeguard in the event that a backwards
incompatible change is introduce to PSBT. While PSBT is designed and
intended to be forwards compatible by allowing parsers to ignore types
that they do not understand, it is possible that at ype is added in the
future which breaks this assumption and it would be unsafe for a type to
be ignored.

Updaters and combiners that need to add a version number to a PSBT
should use the highest version number required. For example, if a
combiner sees two PSBTs for the same transaction, one with version 0,
and the other with version 1, then it should combine them and produce a
PSBT with version 1. If an updater is updating a PSBT and needs to add a
field that is only available in version 1, then it should set the PSBT
version number to 1 unless a version higher than that is already specified.

It is not expected that the version number will ever be used. We try to
make PSBT fields safe to ignore. The version number is only being
included here as a safeguard in the event that breaking compatibilty is
required.


Lastly, I would like to propose the canonical method for mult-byte
types. We designate a specific type to indicate that the type is
multiple bytes. When such types are observed, parsers should move onto
the next byte and interpret that as the type, keeping in mind the number
of bytes that were read in for the type.

I propose that we use 0xFF as this designated type. When a parser sees
an 0xFF value as the type, it reads the next byte as being part of the
type. So two byte types will be of the form 0xFFXX. This method allows
us to do a prefix match in order to quickly identify the type being
used. For types with more bytes, simply use another 0xFF byte. So three
byte types would be of the form 0xFFFFXX, four byte, 0xFFFFFFXX, and so
on. When multi-byte types are specified in the BIP, they should be
specified in this full length form, i.e. two byte types as 0xFFXX.

The same mechanism can be used for the proprietary use types, just with
a different value as the designated multi-byte indicator. For example,
one could use 0xFE as the designated type as that is in the proprietary
types range. Of course any type within the proprietary type range could
be used as the indicator, it is up to the users to determine this
themselves.

While other methods of indicating multiple bytes and lengths may be more
space efficient and allow us to have more types represented in a smaller
space, I am choosing this method because of its simplicity. This is easy
to understand and implement. Furthermore, I do not expect that we will
use so many types. I don't think that we will need to have more than one
byte types for a very long time.


Please let me know your thoughts on these extensions. I will open a PR
to the bips repo to add these to BIP 174 if there are no objections.


Andrew Chow



 permalink raw reply	[|nested]
2019-07-31  1:13 Andrew Chow

  2019-07-31 16:19 ` Dmitry Petukhov
  
From: jan matejek @ 2019-07-31 14:32 UTC (permalink / raw)
  To: Andrew Chow via bitcoin-dev

hello,

On 31. 07. 19 3:13, Andrew Chow via bitcoin-dev wrote:
> Lastly, I would like to propose the canonical method for mult-byte
> types. We designate a specific type to indicate that the type is
> multiple bytes. When such types are observed, parsers should move onto
> the next byte and interpret that as the type, keeping in mind the number
> of bytes that were read in for the type.

why not use Bitcoin compact uint, which most PSBT consumers already
implement?

regards
m.


 permalink raw reply	[|nested]
2019-07-31  1:13 Andrew Chow
  2019-07-31 14:32 ` jan matejek

  2019-07-31 19:16   ` Andrew Chow
  
From: Dmitry Petukhov @ 2019-07-31 16:19 UTC (permalink / raw)
  To: bitcoin-dev

В Wed, 31 Jul 2019 01:13:46 +0000
Andrew Chow via bitcoin-dev <bitcoin-dev@lists•linuxfoundation.org>
wrote:

> Firstly, I would like to propose that some types be reserved for
> proprietary use. These proprietary use types are, in general, for
> private use by individuals and organizations who want to use PSBT in
> their processes. These are usefule when there are additional data that
> they need attached to a PSBT but such data are not useful (or
> available) for the general public.

I think private formats should have at least a basic format: they
should start with a prefix. This way different prviate formats can be
distinguished by this prefix, and there will be no risk of
unintentional confusion.

Private types can start with the size of the prefix, and then
organization can choose any prefix they like, or no prefix, if
the size is of the prefix is 0 (means they are fine with possible
conflicts with other empty-prefix private types)

> Lastly, I would like to propose the canonical method for mult-byte
> types. We designate a specific type to indicate that the type is
> multiple bytes. When such types are observed, parsers should move onto
> the next byte and interpret that as the type, keeping in mind the
> number of bytes that were read in for the type.
> 
> I propose that we use 0xFF as this designated type. When a parser sees
> an 0xFF value as the type, it reads the next byte as being part of the
> type. So two byte types will be of the form 0xFFXX. This method allows
> us to do a prefix match in order to quickly identify the type being
> used. For types with more bytes, simply use another 0xFF byte. So
> three byte types would be of the form 0xFFFFXX, four byte,
> 0xFFFFFFXX, and so on. When multi-byte types are specified in the
> BIP, they should be specified in this full length form, i.e. two byte
> types as 0xFFXX.

Why not just say that the types should be encoded as 'compact size
unsigned integer' ? This format for variable length integer encoding is
already used in the BIP for other fields, and thus will not add any
additional complexity to the parsing.


 permalink raw reply	[|nested]
2019-07-31 16:19 ` Dmitry Petukhov

  
From: Andrew Chow @ 2019-07-31 19:16 UTC (permalink / raw)
  To: Dmitry Petukhov, bitcoin-dev

Hi,

On 7/31/19 12:19 PM, Dmitry Petukhov wrote:
> 
> I think private formats should have at least a basic format: they
> should start with a prefix. This way different prviate formats can be
> distinguished by this prefix, and there will be no risk of
> unintentional confusion.
> 
> Private types can start with the size of the prefix, and then
> organization can choose any prefix they like, or no prefix, if
> the size is of the prefix is 0 (means they are fine with possible
> conflicts with other empty-prefix private types)
> 

I don't think that should something that is required for people to do,
but perhaps it can be something that is strongly recommended and
suggested in the BIP itself.

> 
> Why not just say that the types should be encoded as 'compact size
> unsigned integer' ? This format for variable length integer encoding is
> already used in the BIP for other fields, and thus will not add any
> additional complexity to the parsing.
> 

On 7/31/19 10:32 AM, jan matejek via bitcoin-dev wrote:>
>
> why not use Bitcoin compact uint, which most PSBT consumers already
> implement?
>

There are a few issues with using a compact size uint. The main issue is
that it doesn't translate well to the proprietary use types. If we used
CSUint for the type, then all of type values for proprietary use need to
be reserved instead of allowing them to be infinitely expanded from the
initial set of proprietary use types.

There is also the fact that CSUints are malleable as the same value can
be represented in many different ways, just with different amounts of
leading zeroes. But I suppose that isn't really that big of an issue.

I am not opposed to using a CSUint, I just felt that it made things a
bit harder and was unnecessary.


Andrew



 permalink raw reply	[|nested]


  2019-08-01 17:57   ` Andrew Chow
  
From: Stepan Snigirev @ 2019-08-01 13:50 UTC (permalink / raw)
  To: bitcoin-dev

[-- Attachment #1: Type: text/plain, Size: 491 bytes --]

> why not use Bitcoin compact uint, which most PSBT consumers already
implement?

I totally agree with that, compact int parsing is already implemented in
all bitcoin applications, wallets and libraries. Also, for certain (mb
proprietary) applications I will be willing to use multi-byte keys where
the first byte defines the application type and next bytes define
application-specific fields.

I would suggest to set proprietary bytes to 0xF0-0xFC or to 0xE0-0xEF then
(E for Enterprise?).

[-- Attachment #2: Type: text/html, Size: 574 bytes --]

 permalink raw reply	[|nested]
2019-08-01 13:50 ` [bitcoin-dev] Proposed Extensions to BIP 174 for Future Extensibility Stepan Snigirev

  2019-08-01 19:01     ` Andrew Chow
  
From: Andrew Chow @ 2019-08-01 17:57 UTC (permalink / raw)
  To: bitcoin-dev

It seems like the consensus is that we should use Compact Size Unsigned
Integers for the field types, so we will do that. The types will be
minimally encoded CSUints.

For the proprietary types, I will use Dimitry's and Andrew Poelstra's
(https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-March/016713.html)
suggestion. There will be one proprietary type, 0xFC. This will be
followed by a variable length string that is a vendor specific prefix
that serves as a unique identifier to help with preventing usage of
proprietary types outside of private contexts. This will then be
followed by the actual type for the data, as defined by the user of the
proprietary type.

The prefix will just be a string, prefixed with a compact size unsigned
integer. If no prefix is wanted, then a single 0x00 byte can be used.

For public software, there is no need to handle these proprietary types
at all so they do not need to check the string or the data type. It is
not necessary to enforce the above rules about proprietary types except
that they use the proprietary type value.


Andrew Chow



 permalink raw reply	[|nested]
2019-08-01 17:57   ` Andrew Chow

  2019-08-02  9:18       ` Dmitry Petukhov
  
From: Andrew Chow @ 2019-08-01 19:01 UTC (permalink / raw)
  To: bitcoin-dev, Dmitry Petukhov

I spoke to some people OOB and they said that they didn't really like
the idea of having a prefix string (partially because they've already
implemented some proprietary types by simply squatting on unused types).
Matching the prefix string adds additional complexity to the parser
code. The main concern is that people won't want to actually follow the
spec for proprietary types and instead just use some unused type value.
So I think instead we should do:

{0xFC}|{private_type}

and the prefix string can be optional (and strongly recommended) after that.

Since public parsers won't really be enforcing the rules for proprietary
types, I don't think it really makes sense to specify and enforce how
they should be. After all, the key is really just an opaque data blob.

In the same vein, it would probably be useful to allow multiple types
for proprietary use as originally proposed to make implementation of
these easier. If more type are needed, then the private type
construction can be used.


Andrew

On 8/1/19 1:57 PM, Andrew Chow wrote:
> 
> It seems like the consensus is that we should use Compact Size Unsigned
> Integers for the field types, so we will do that. The types will be
> minimally encoded CSUints.
> 
> For the proprietary types, I will use Dimitry's and Andrew Poelstra's
> (https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-March/016713.html)
> suggestion. There will be one proprietary type, 0xFC. This will be
> followed by a variable length string that is a vendor specific prefix
> that serves as a unique identifier to help with preventing usage of
> proprietary types outside of private contexts. This will then be
> followed by the actual type for the data, as defined by the user of the
> proprietary type.
> 
> The prefix will just be a string, prefixed with a compact size unsigned
> integer. If no prefix is wanted, then a single 0x00 byte can be used.
> 
> For public software, there is no need to handle these proprietary types
> at all so they do not need to check the string or the data type. It is
> not necessary to enforce the above rules about proprietary types except
> that they use the proprietary type value.
> 
> 
> Andrew Chow
> 



 permalink raw reply	[|nested]
2019-08-01 19:01     ` Andrew Chow

  2019-08-04  0:15         ` Jonathan Underwood
  
From: Dmitry Petukhov @ 2019-08-02  9:18 UTC (permalink / raw)
  To: Andrew Chow;  bitcoin-dev

В Thu, 01 Aug 2019 19:01:06 +0000
Andrew Chow <achow101-lists@achow101•com> wrote:

> I spoke to some people OOB and they said that they didn't really like
> the idea of having a prefix string (partially because they've already
> implemented some proprietary types by simply squatting on unused
> types). Matching the prefix string adds additional complexity to the
> parser code.

I do not oppose the idea of "{0xFC}|{private_type}" strongly, but I
would like to note that for those who do not want to deal with
additional complexity of handling a prefixed string, they can simply
not use it at all. Since this is a private construction, and their
private format specifies 'no prefix', they can just ignore everything
that does not start with "{0xFC}|{0x00}", thus any further complexity
regarding the prefix is also ignored. The only added complexity is one
condition check: second_byte_of_the_key != 0 

My other argument for conflict-avoidance prefix as a first thing after
0xFC is that the set of future users of PSBT and private types is
most likely much larger than the current set of those who already
implemented proprietary types on their own, and thus the overall benefit
for the whole industry will likely be bigger when 'i do not want
conflict avoidance' decision have to be explicit, by setting the prefix
to 0x00, and the set of possible conflicting types are limited only to
those entities that made this explicit decision.

Regarding the 'squatted' types, it seems to me that this only matters
in the discussed context if they squatted on 0xFC type in particular.
In other cases, they will need to implement changes anyway, to be
compatible with the BIP. Maybe they could consider that one additional
condition check is a small burden, and maybe they can tolerate that,
for the benefit of reducing possibility of interoperability problems
between other future PSBT/private types implementors.



 permalink raw reply	[|nested]
2019-08-02  9:18       ` Dmitry Petukhov

  
From: Jonathan Underwood @ 2019-08-04  0:15 UTC ( / )
  To: Bitcoin Protocol Discussion



My two cents:

1. Reserved types are awesome.
2. Varint for type is awesome.
3. BIP174 should specify a specific type for all (global, input, and
output) which means "see the BIP numbered in the next byte" so we can have
some sort of BIP43-ish system for BIP174... POR COMMITMENT and my current
signature protocol proposal should go in there.

More like three cents, but you get the idea.

I'll keep an eye on the bips repo. If someone wants to ping me once things
settle down I'll implement it.

Thanks,
Jon

2019年8月2日(金) 20:34 Dmitry Petukhov via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org>:

> В Thu, 01 Aug 2019 19:01:06 +0000
> Andrew Chow <achow101-lists@achow101•com> wrote:
>
> > I spoke to some people OOB and they said that they didn't really like
> > the idea of having a prefix string (partially because they've already
> > implemented some proprietary types by simply squatting on unused
> > types). Matching the prefix string adds additional complexity to the
> > parser code.
>
> I do not oppose the idea of "{0xFC}|{private_type}" strongly, but I
> would like to note that for those who do not want to deal with
> additional complexity of handling a prefixed string, they can simply
> not use it at all. Since this is a private construction, and their
> private format specifies 'no prefix', they can just ignore everything
> that does not start with "{0xFC}|{0x00}", thus any further complexity
> regarding the prefix is also ignored. The only added complexity is one
> condition check: second_byte_of_the_key != 0
>
> My other argument for conflict-avoidance prefix as a first thing after
> 0xFC is that the set of future users of PSBT and private types is
> most likely much larger than the current set of those who already
> implemented proprietary types on their own, and thus the overall benefit
> for the whole industry will likely be bigger when 'i do not want
> conflict avoidance' decision have to be explicit, by setting the prefix
> to 0x00, and the set of possible conflicting types are limited only to
> those entities that made this explicit decision.
>
> Regarding the 'squatted' types, it seems to me that this only matters
> in the discussed context if they squatted on 0xFC type in particular.
> In other cases, they will need to implement changes anyway, to be
> compatible with the BIP. Maybe they could consider that one additional
> condition check is a small burden, and maybe they can tolerate that,
> for the benefit of reducing possibility of interoperability problems
> between other future PSBT/private types implementors.
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>


-- 
-----------------
Jonathan Underwood
ビットバンク社 チーフビットコインオフィサー
-----------------

暗号化したメッセージをお送りの方は下記の公開鍵をご利用下さい。

指紋: 0xCE5EA9476DE7D3E45EBC3FDAD998682F3590FEA3