The Tragic Tale of BIP30

The Tragic Tale of BIP30
2025-04-27 18:20 ` Luke Dashjr
                   ` (2 more replies)
  
From: Ruben Somsen @ 2025-04-27 16:45 UTC (permalink / raw)
  To: bitcoindev

[-- Attachment #1: Type: text/plain, Size: 5563 bytes --]

Markdown version:
https://gist.github.com/RubenSomsen/a02b9071bf81b922dcc9edea7d810b7c

### Introduction

In my recent exploration of [SwiftSync][1], I came to the realization that
[BIP30][2] has an unresolved consensus bug. It seems this bug can't be
triggered without a reorg back to 2010, so its seriousness is debatable. We
currently have checkpoints up to 2013, preventing such a reorg. Once we
fully [remove the checkpoints][3], the bug becomes theoretically (not
practically) exploitable.

BIP30 is also a bit of an odd duck in terms of consensus checks in that it
involves the entire UTXO set without being related to the spending of
inputs. This is inefficient, and complicates the implementation of
alternative validation methods such as utreexo, SwiftSync and quite
possibly ZKP systems such as ZeroSync. It would be nice if we could sunset
BIP30 altogether.

Without necessarily advocating for action (the status quo seems quite
tenable), I'd like to lay out possible solutions for both and open up the
discussion.

### 1. Consensus bug

There are two duplicate transactions that BIP30 treats like exceptions. The
last duplicate was the coinbase transaction in block 91880. When this
transaction gets processed, the coinbase transaction in block 91722 is
overwritten. The other instance occurs between these two blocks (91812,
91842).

The problem occurs when we reorg back to a point between block 91880 and
91722. When we rewind the blockchain, previously created outputs get
removed from the UTXO set again. As a result, the overwritten output
disappears from the UTXO set completely. A node that never witnessed the
reorg, however, will still have the UTXO in its set. If subsequently the
UTXO is ever spent, this would result in a fork.

#### Solution A

We could enforce that no reorg can take place between block 91722 and 91880
- you'd either have to reorg all of them, or none. This ensures both
reorged and fresh nodes will not have the problematic outputs in their UTXO
set. Considering this is only ~160 blocks at the low mining difficulty of
2010, this wouldn't be a big constraint.

#### Solution B

When discussing my findings with Sjors Provoost, he pointed out that the
removal of the checkpoints (which can be seen as a hard fork) [that is
being considered][3] also presents a window of opportunity to change the
pre-checkpoint consensus rules - we could fix the bug by no longer removing
the coinbase transaction in case of a reorg of block 91880 and 91842. Aside
from that, Sjors' observation also opens up the question whether there are
other pre-2013 consensus changes we'd want to consider making.

### 2. Sunsetting BIP30's UTXO set check

BIP30 is currently active from genesis until [BIP34][4] activates at block
height 227931 (March 2013). If this block is reorged out, BIP30 remains
active indefinitely. BIP34 has issues of its own that are being addressed
in the [Consensus Cleanup BIP][5] - you could go and read that, I won't go
into too much detail here.

Technically, BIP30 only prevents duplicate _unspent_ outputs. It does this
by checking for each output whether it's already in the UTXO set (this is
the inefficient part), and rejecting the block if it is. The two 2010
duplicates are hard-coded in as exceptions. Under these rules, spending an
output and recreating it is allowed. However, it seems this never occurred.

One last point to address is why BIP34 gets deactivated if block 227931 is
reorged out. The reason for this is because otherwise it'd open the door to
possibly creating outputs prior to BIP34's activation that will conflict
with BIP34's rules for ensuring coinbase transaction uniqueness (the exact
issue the Consensus Cleanup is seeking to resolve).

Ideally, it'd be nice to be able to sunset the BIP30 UTXO set check
completely, ensuring it's no longer required, even in case of a reorg.

#### Solution

Given that we have no duplicates, barring the two exceptions, we could
replace the inefficient BIP30 UTXO set check with a coinbase uniqueness
check. We simply cache the coinbase TXIDs and ensure there are no
duplicates. Doing this until block 227931 results in a modest ~7MB cache.
However, BIP30 might not deactivate, in which case we'd have an
ever-growing cache. This is solvable as follows.

Aside from checking for coinbase uniqueness, we could also check that the
coinbase will not conflict with any future coinbases (i.e. not conflict
with BIP34 as well as the Consensus Cleanup BIP). This ensures BIP34 can
simply activate at block height 227931, regardless of whether there's a
reorg.

### In closing

These were some of the issues with BIP30 and possible solutions. Regardless
of whether we choose to take action, this write-up will serve as a
reference. Thanks to Antoine Poinsot, Pieter Wuille, and Sjors Provoost for
the discussions prior to publishing.

-- Ruben Somsen


[1]: https://gist.github.com/RubenSomsen/a61a37d14182ccd78760e477c78133cd
[2]: https://github.com/bitcoin/bips/blob/master/bip-0030.mediawiki
[3]: https://github.com/bitcoin/bitcoin/pull/31649
[4]: https://github.com/bitcoin/bips/blob/master/bip-0034.mediawiki
[5]: https://github.com/bitcoin/bips/pull/1800

-- 
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups•com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/CAPv7TjZTWhgzzdps3vb0YoU3EYJwThDFhNLkf4XmmdfhbORTaw%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 6540 bytes --]

 permalink raw reply	[|nested]
2025-04-27 18:20 ` 
                   ` 
  
From: Ruben Somsen @ 2025-04-27 16:45 UTC (permalink / raw)
  To: bitcoindev

[-- Attachment #1: Type: text/plain, Size: 5563 bytes --]

Markdown version:
https://gist.github.com/RubenSomsen/a02b9071bf81b922dcc9edea7d810b7c

### Introduction

In my recent exploration of [SwiftSync][1], I came to the realization that
[BIP30][2] has an unresolved consensus bug. It seems this bug can't be
triggered without a reorg back to 2010, so its seriousness is debatable. We
currently have checkpoints up to 2013, preventing such a reorg. Once we
fully [remove the checkpoints][3], the bug becomes theoretically (not
practically) exploitable.

BIP30 is also a bit of an odd duck in terms of consensus checks in that it
involves the entire UTXO set without being related to the spending of
inputs. This is inefficient, and complicates the implementation of
alternative validation methods such as utreexo, SwiftSync and quite
possibly ZKP systems such as ZeroSync. It would be nice if we could sunset
BIP30 altogether.

Without necessarily advocating for action (the status quo seems quite
tenable), I'd like to lay out possible solutions for both and open up the
discussion.

### 1. Consensus bug

There are two duplicate transactions that BIP30 treats like exceptions. The
last duplicate was the coinbase transaction in block 91880. When this
transaction gets processed, the coinbase transaction in block 91722 is
overwritten. The other instance occurs between these two blocks (91812,
91842).

The problem occurs when we reorg back to a point between block 91880 and
91722. When we rewind the blockchain, previously created outputs get
removed from the UTXO set again. As a result, the overwritten output
disappears from the UTXO set completely. A node that never witnessed the
reorg, however, will still have the UTXO in its set. If subsequently the
UTXO is ever spent, this would result in a fork.

#### Solution A

We could enforce that no reorg can take place between block 91722 and 91880
- you'd either have to reorg all of them, or none. This ensures both
reorged and fresh nodes will not have the problematic outputs in their UTXO
set. Considering this is only ~160 blocks at the low mining difficulty of
2010, this wouldn't be a big constraint.

#### Solution B

When discussing my findings with Sjors Provoost, he pointed out that the
removal of the checkpoints (which can be seen as a hard fork) [that is
being considered][3] also presents a window of opportunity to change the
pre-checkpoint consensus rules - we could fix the bug by no longer removing
the coinbase transaction in case of a reorg of block 91880 and 91842. Aside
from that, Sjors' observation also opens up the question whether there are
other pre-2013 consensus changes we'd want to consider making.

### 2. Sunsetting BIP30's UTXO set check

BIP30 is currently active from genesis until [BIP34][4] activates at block
height 227931 (March 2013). If this block is reorged out, BIP30 remains
active indefinitely. BIP34 has issues of its own that are being addressed
in the [Consensus Cleanup BIP][5] - you could go and read that, I won't go
into too much detail here.

Technically, BIP30 only prevents duplicate _unspent_ outputs. It does this
by checking for each output whether it's already in the UTXO set (this is
the inefficient part), and rejecting the block if it is. The two 2010
duplicates are hard-coded in as exceptions. Under these rules, spending an
output and recreating it is allowed. However, it seems this never occurred.

One last point to address is why BIP34 gets deactivated if block 227931 is
reorged out. The reason for this is because otherwise it'd open the door to
possibly creating outputs prior to BIP34's activation that will conflict
with BIP34's rules for ensuring coinbase transaction uniqueness (the exact
issue the Consensus Cleanup is seeking to resolve).

Ideally, it'd be nice to be able to sunset the BIP30 UTXO set check
completely, ensuring it's no longer required, even in case of a reorg.

#### Solution

Given that we have no duplicates, barring the two exceptions, we could
replace the inefficient BIP30 UTXO set check with a coinbase uniqueness
check. We simply cache the coinbase TXIDs and ensure there are no
duplicates. Doing this until block 227931 results in a modest ~7MB cache.
However, BIP30 might not deactivate, in which case we'd have an
ever-growing cache. This is solvable as follows.

Aside from checking for coinbase uniqueness, we could also check that the
coinbase will not conflict with any future coinbases (i.e. not conflict
with BIP34 as well as the Consensus Cleanup BIP). This ensures BIP34 can
simply activate at block height 227931, regardless of whether there's a
reorg.

### In closing

These were some of the issues with BIP30 and possible solutions. Regardless
of whether we choose to take action, this write-up will serve as a
reference. Thanks to Antoine Poinsot, Pieter Wuille, and Sjors Provoost for
the discussions prior to publishing.

-- Ruben Somsen


[1]: https://gist.github.com/RubenSomsen/a61a37d14182ccd78760e477c78133cd
[2]: https://github.com/bitcoin/bips/blob/master/bip-0030.mediawiki
[3]: https://github.com/bitcoin/bitcoin/pull/31649
[4]: https://github.com/bitcoin/bips/blob/master/bip-0034.mediawiki
[5]: https://github.com/bitcoin/bips/pull/1800

-- 
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups•com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/CAPv7TjZTWhgzzdps3vb0YoU3EYJwThDFhNLkf4XmmdfhbORTaw%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 6540 bytes --]

 permalink raw reply	[|nested]
2025-04-27 16:45 [bitcoindev] The Tragic Tale of BIP30 Ruben Somsen
  2025-04-27 18:20 ` Luke Dashjr

  2025-04-27 21:01   ` eric
  2025-04-28 11:48 ` Sjors Provoost
  
From: eric @ 2025-04-27 18:30 UTC (permalink / raw)
  To: 'Ruben Somsen', bitcoindev

> The problem occurs when we reorg back to a point between block 91880 and
> 91722. When we rewind the blockchain, previously created outputs get
> removed from the UTXO set again. 

Consider that a UTXO set (accumulator) is an implementation detail. One of its underlying assumptions is that a given txid cannot repeat in confirmed blocks. This assumption did not hold, and the bip30 workaround for the two exceptions failed to consider the effect of the above reorg given a UTXO set design. Instead of removing the second instance, it removes all instances.

> we could fix the bug by no longer removing the coinbase transaction in case of a reorg of block 91880 and 91842.

IMO this would be the most reasonable resolution, as it would produce the outcome that in fact exists independent of the UTXO set. The previous blocks still actually have the first instance of each duplicated coinbase.

This is also inconsequential from a performance standpoint. It only affects reorganization (infrequent), and only proceeds for these two specific blocks (rare). Furthermore these two specific blocks are already exceptions and the implementation is trivial.

> However, it seems this never occurred.

Correct, only the exception coinbases have been duplicated in the current strong chain.

> Aside from checking for coinbase uniqueness, we could also check that the
> coinbase will not conflict with any future coinbases (i.e. not conflict with
> BIP34 as well as the Consensus Cleanup BIP).

The relationship between BIP34 and BIP30 is also a bit sordid, but the presumption is that the Consensus Cleanup would resolve the existing flaw in BIP34, and that the combination would effectively obsolete BIP30. Under the assumption that this does in fact produce the intended outcome - that BIP34 (presently) Consensus Cleanup (forever) makes further coinbase duplication impossible, BIP30 can remain deactivated to the extent that these are active. Nothing additional is required to avoid the presumably inefficient BIP30 checks.

> Once we fully [remove the checkpoints][3], the bug becomes theoretically (not practically) exploitable.

I would never advocate for adding more, but I'm not aware of any compelling argument to hard fork out the existing checkpoints. The top checkpoint is consensus for over 11 years and materially reduces the validation cost of 295,000 blocks.

> Doing this until block 227931 results in a modest ~7MB cache. However,
> BIP30 might not deactivate, in which case we'd have an ever-growing cache.
> This is solvable as follows....

This is a consequence of the presumed removal of checkpoints above BIP34 activation height. IOW, removing the checkpoints makes it necessary to validate BIP30 until BIP34 activates (block 227,931). The obvious solution to this problem is to not create the problem in the first place.

e

> -----Original Message-----
> From: bitcoindev@googlegroups.com <bitcoindev@googlegroups.com> On
> Behalf Of Ruben Somsen
> Sent: Sunday, April 27, 2025 12:45 PM
> To: bitcoindev@googlegroups.com
> Subject: [bitcoindev] The Tragic Tale of BIP30
> 
> Markdown version:
> https://gist.github.com/RubenSomsen/a02b9071bf81b922dcc9edea7d810b
> 7c
> 
> ### Introduction
> 
> In my recent exploration of [SwiftSync][1], I came to the realization that
> [BIP30][2] has an unresolved consensus bug. It seems this bug can't be
> triggered without a reorg back to 2010, so its seriousness is debatable. We
> currently have checkpoints up to 2013, preventing such a reorg. Once we fully
> [remove the checkpoints][3], the bug becomes theoretically (not practically)
> exploitable.
> 
> BIP30 is also a bit of an odd duck in terms of consensus checks in that it
> involves the entire UTXO set without being related to the spending of inputs.
> This is inefficient, and complicates the implementation of alternative validation
> methods such as utreexo, SwiftSync and quite possibly ZKP systems such as
> ZeroSync. It would be nice if we could sunset BIP30 altogether.
> 
> Without necessarily advocating for action (the status quo seems quite
> tenable), I'd like to lay out possible solutions for both and open up the
> discussion.
> 
> ### 1. Consensus bug
> 
> There are two duplicate transactions that BIP30 treats like exceptions. The last
> duplicate was the coinbase transaction in block 91880. When this transaction
> gets processed, the coinbase transaction in block 91722 is overwritten. The
> other instance occurs between these two blocks (91812, 91842).
> 
> The problem occurs when we reorg back to a point between block 91880 and
> 91722. When we rewind the blockchain, previously created outputs get
> removed from the UTXO set again. As a result, the overwritten output
> disappears from the UTXO set completely. A node that never witnessed the
> reorg, however, will still have the UTXO in its set. If subsequently the UTXO is
> ever spent, this would result in a fork.
> 
> #### Solution A
> 
> We could enforce that no reorg can take place between block 91722 and
> 91880 - you'd either have to reorg all of them, or none. This ensures both
> reorged and fresh nodes will not have the problematic outputs in their UTXO
> set. Considering this is only ~160 blocks at the low mining difficulty of 2010,
> this wouldn't be a big constraint.
> 
> #### Solution B
> 
> When discussing my findings with Sjors Provoost, he pointed out that the
> removal of the checkpoints (which can be seen as a hard fork) [that is being
> considered][3] also presents a window of opportunity to change the pre-
> checkpoint consensus rules - we could fix the bug by no longer removing the
> coinbase transaction in case of a reorg of block 91880 and 91842. Aside from
> that, Sjors' observation also opens up the question whether there are other
> pre-2013 consensus changes we'd want to consider making.
> 
> ### 2. Sunsetting BIP30's UTXO set check
> 
> BIP30 is currently active from genesis until [BIP34][4] activates at block height
> 227931 (March 2013). If this block is reorged out, BIP30 remains active
> indefinitely. BIP34 has issues of its own that are being addressed in the
> [Consensus Cleanup BIP][5] - you could go and read that, I won't go into too
> much detail here.
> 
> Technically, BIP30 only prevents duplicate _unspent_ outputs. It does this by
> checking for each output whether it's already in the UTXO set (this is the
> inefficient part), and rejecting the block if it is. The two 2010 duplicates are
> hard-coded in as exceptions. Under these rules, spending an output and
> recreating it is allowed. However, it seems this never occurred.
> 
> One last point to address is why BIP34 gets deactivated if block 227931 is
> reorged out. The reason for this is because otherwise it'd open the door to
> possibly creating outputs prior to BIP34's activation that will conflict with
> BIP34's rules for ensuring coinbase transaction uniqueness (the exact issue the
> Consensus Cleanup is seeking to resolve).
> 
> Ideally, it'd be nice to be able to sunset the BIP30 UTXO set check completely,
> ensuring it's no longer required, even in case of a reorg.
> 
> #### Solution
> 
> Given that we have no duplicates, barring the two exceptions, we could
> replace the inefficient BIP30 UTXO set check with a coinbase uniqueness
> check. We simply cache the coinbase TXIDs and ensure there are no duplicates.
> Doing this until block 227931 results in a modest ~7MB cache. However,
> BIP30 might not deactivate, in which case we'd have an ever-growing cache.
> This is solvable as follows.
> 
> Aside from checking for coinbase uniqueness, we could also check that the
> coinbase will not conflict with any future coinbases (i.e. not conflict with
> BIP34 as well as the Consensus Cleanup BIP). This ensures BIP34 can simply
> activate at block height 227931, regardless of whether there's a reorg.
> 
> ### In closing
> 
> These were some of the issues with BIP30 and possible solutions. Regardless
> of whether we choose to take action, this write-up will serve as a reference.
> Thanks to Antoine Poinsot, Pieter Wuille, and Sjors Provoost for the
> discussions prior to publishing.
> 
> -- Ruben Somsen
> 
> 
> [1]:
> https://gist.github.com/RubenSomsen/a61a37d14182ccd78760e477c7813
> 3cd
> [2]: https://github.com/bitcoin/bips/blob/master/bip-0030.mediawiki
> [3]: https://github.com/bitcoin/bitcoin/pull/31649
> [4]: https://github.com/bitcoin/bips/blob/master/bip-0034.mediawiki
> [5]: https://github.com/bitcoin/bips/pull/1800
> 
> --
> You received this message because you are subscribed to the Google Groups
> "Bitcoin Development Mailing List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bitcoindev+unsubscribe@googlegroups•com
> <mailto:bitcoindev+unsubscribe@googlegroups•com> .
> To view this discussion visit
> https://groups.google.com/d/msgid/bitcoindev/CAPv7TjZTWhgzzdps3vb0Yo
> U3EYJwThDFhNLkf4XmmdfhbORTaw%40mail.gmail.com
> <https://groups.google.com/d/msgid/bitcoindev/CAPv7TjZTWhgzzdps3vb0Y
> oU3EYJwThDFhNLkf4XmmdfhbORTaw%40mail.gmail.com?utm_medium=em
> ail&utm_source=footer> .


-- 
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups•com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/002201dbb7a2%2474676640%245d3632c0%24%40voskuil.org.


 permalink raw reply	[|nested]
2025-04-27 18:30 ` eric

  
From: eric @ 2025-04-27 21:01 UTC (permalink / raw)
  To: 'Ruben Somsen', bitcoindev

> The top checkpoint is consensus for over 11 years

Correction: the block of the top checkpoint has been confirmed for over 11 years.

e

> -----Original Message-----
> From: eric@voskuil•org <eric@voskuil•org>
> Sent: Sunday, April 27, 2025 2:30 PM
> To: 'Ruben Somsen' <rsomsen@gmail•com>; bitcoindev@googlegroups.com
> Subject: RE: [bitcoindev] The Tragic Tale of BIP30
> 
> > The problem occurs when we reorg back to a point between block 91880
> > and 91722. When we rewind the blockchain, previously created outputs
> > get removed from the UTXO set again.
> 
> Consider that a UTXO set (accumulator) is an implementation detail. One of its
> underlying assumptions is that a given txid cannot repeat in confirmed blocks.
> This assumption did not hold, and the bip30 workaround for the two
> exceptions failed to consider the effect of the above reorg given a UTXO set
> design. Instead of removing the second instance, it removes all instances.
> 
> > we could fix the bug by no longer removing the coinbase transaction in case
> of a reorg of block 91880 and 91842.
> 
> IMO this would be the most reasonable resolution, as it would produce the
> outcome that in fact exists independent of the UTXO set. The previous blocks
> still actually have the first instance of each duplicated coinbase.
> 
> This is also inconsequential from a performance standpoint. It only affects
> reorganization (infrequent), and only proceeds for these two specific blocks
> (rare). Furthermore these two specific blocks are already exceptions and the
> implementation is trivial.
> 
> > However, it seems this never occurred.
> 
> Correct, only the exception coinbases have been duplicated in the current
> strong chain.
> 
> > Aside from checking for coinbase uniqueness, we could also check that
> > the coinbase will not conflict with any future coinbases (i.e. not
> > conflict with
> > BIP34 as well as the Consensus Cleanup BIP).
> 
> The relationship between BIP34 and BIP30 is also a bit sordid, but the
> presumption is that the Consensus Cleanup would resolve the existing flaw in
> BIP34, and that the combination would effectively obsolete BIP30. Under the
> assumption that this does in fact produce the intended outcome - that BIP34
> (presently) Consensus Cleanup (forever) makes further coinbase duplication
> impossible, BIP30 can remain deactivated to the extent that these are active.
> Nothing additional is required to avoid the presumably inefficient BIP30
> checks.
> 
> > Once we fully [remove the checkpoints][3], the bug becomes theoretically
> (not practically) exploitable.
> 
> I would never advocate for adding more, but I'm not aware of any compelling
> argument to hard fork out the existing checkpoints. The top checkpoint is
> consensus for over 11 years and materially reduces the validation cost of
> 295,000 blocks.
> 
> > Doing this until block 227931 results in a modest ~7MB cache. However,
> > BIP30 might not deactivate, in which case we'd have an ever-growing cache.
> > This is solvable as follows....
> 
> This is a consequence of the presumed removal of checkpoints above BIP34
> activation height. IOW, removing the checkpoints makes it necessary to
> validate BIP30 until BIP34 activates (block 227,931). The obvious solution to
> this problem is to not create the problem in the first place.
> 
> e
> 
> > -----Original Message-----
> > From: bitcoindev@googlegroups.com <bitcoindev@googlegroups.com> On
> > Behalf Of Ruben Somsen
> > Sent: Sunday, April 27, 2025 12:45 PM
> > To: bitcoindev@googlegroups.com
> > Subject: [bitcoindev] The Tragic Tale of BIP30
> >
> > Markdown version:
> >
> https://gist.github.com/RubenSomsen/a02b9071bf81b922dcc9edea7d810b
> > 7c
> >
> > ### Introduction
> >
> > In my recent exploration of [SwiftSync][1], I came to the realization
> > that [BIP30][2] has an unresolved consensus bug. It seems this bug
> > can't be triggered without a reorg back to 2010, so its seriousness is
> > debatable. We currently have checkpoints up to 2013, preventing such a
> > reorg. Once we fully [remove the checkpoints][3], the bug becomes
> > theoretically (not practically) exploitable.
> >
> > BIP30 is also a bit of an odd duck in terms of consensus checks in
> > that it involves the entire UTXO set without being related to the spending of
> inputs.
> > This is inefficient, and complicates the implementation of alternative
> > validation methods such as utreexo, SwiftSync and quite possibly ZKP
> > systems such as ZeroSync. It would be nice if we could sunset BIP30
> altogether.
> >
> > Without necessarily advocating for action (the status quo seems quite
> > tenable), I'd like to lay out possible solutions for both and open up
> > the discussion.
> >
> > ### 1. Consensus bug
> >
> > There are two duplicate transactions that BIP30 treats like
> > exceptions. The last duplicate was the coinbase transaction in block
> > 91880. When this transaction gets processed, the coinbase transaction
> > in block 91722 is overwritten. The other instance occurs between these two
> blocks (91812, 91842).
> >
> > The problem occurs when we reorg back to a point between block 91880
> > and 91722. When we rewind the blockchain, previously created outputs
> > get removed from the UTXO set again. As a result, the overwritten
> > output disappears from the UTXO set completely. A node that never
> > witnessed the reorg, however, will still have the UTXO in its set. If
> > subsequently the UTXO is ever spent, this would result in a fork.
> >
> > #### Solution A
> >
> > We could enforce that no reorg can take place between block 91722 and
> > 91880 - you'd either have to reorg all of them, or none. This ensures
> > both reorged and fresh nodes will not have the problematic outputs in
> > their UTXO set. Considering this is only ~160 blocks at the low mining
> > difficulty of 2010, this wouldn't be a big constraint.
> >
> > #### Solution B
> >
> > When discussing my findings with Sjors Provoost, he pointed out that
> > the removal of the checkpoints (which can be seen as a hard fork)
> > [that is being considered][3] also presents a window of opportunity to
> > change the pre- checkpoint consensus rules - we could fix the bug by
> > no longer removing the coinbase transaction in case of a reorg of
> > block 91880 and 91842. Aside from that, Sjors' observation also opens
> > up the question whether there are other
> > pre-2013 consensus changes we'd want to consider making.
> >
> > ### 2. Sunsetting BIP30's UTXO set check
> >
> > BIP30 is currently active from genesis until [BIP34][4] activates at
> > block height
> > 227931 (March 2013). If this block is reorged out, BIP30 remains
> > active indefinitely. BIP34 has issues of its own that are being
> > addressed in the [Consensus Cleanup BIP][5] - you could go and read
> > that, I won't go into too much detail here.
> >
> > Technically, BIP30 only prevents duplicate _unspent_ outputs. It does
> > this by checking for each output whether it's already in the UTXO set
> > (this is the inefficient part), and rejecting the block if it is. The
> > two 2010 duplicates are hard-coded in as exceptions. Under these
> > rules, spending an output and recreating it is allowed. However, it seems this
> never occurred.
> >
> > One last point to address is why BIP34 gets deactivated if block
> > 227931 is reorged out. The reason for this is because otherwise it'd
> > open the door to possibly creating outputs prior to BIP34's activation
> > that will conflict with BIP34's rules for ensuring coinbase
> > transaction uniqueness (the exact issue the Consensus Cleanup is seeking to
> resolve).
> >
> > Ideally, it'd be nice to be able to sunset the BIP30 UTXO set check
> > completely, ensuring it's no longer required, even in case of a reorg.
> >
> > #### Solution
> >
> > Given that we have no duplicates, barring the two exceptions, we could
> > replace the inefficient BIP30 UTXO set check with a coinbase
> > uniqueness check. We simply cache the coinbase TXIDs and ensure there are
> no duplicates.
> > Doing this until block 227931 results in a modest ~7MB cache. However,
> > BIP30 might not deactivate, in which case we'd have an ever-growing cache.
> > This is solvable as follows.
> >
> > Aside from checking for coinbase uniqueness, we could also check that
> > the coinbase will not conflict with any future coinbases (i.e. not
> > conflict with
> > BIP34 as well as the Consensus Cleanup BIP). This ensures BIP34 can
> > simply activate at block height 227931, regardless of whether there's a
> reorg.
> >
> > ### In closing
> >
> > These were some of the issues with BIP30 and possible solutions.
> > Regardless of whether we choose to take action, this write-up will serve as a
> reference.
> > Thanks to Antoine Poinsot, Pieter Wuille, and Sjors Provoost for the
> > discussions prior to publishing.
> >
> > -- Ruben Somsen
> >
> >
> > [1]:
> >
> https://gist.github.com/RubenSomsen/a61a37d14182ccd78760e477c7813
> > 3cd
> > [2]: https://github.com/bitcoin/bips/blob/master/bip-0030.mediawiki
> > [3]: https://github.com/bitcoin/bitcoin/pull/31649
> > [4]: https://github.com/bitcoin/bips/blob/master/bip-0034.mediawiki
> > [5]: https://github.com/bitcoin/bips/pull/1800
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Bitcoin Development Mailing List" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> > an email to bitcoindev+unsubscribe@googlegroups•com
> > <mailto:bitcoindev+unsubscribe@googlegroups•com> .
> > To view this discussion visit
> >
> https://groups.google.com/d/msgid/bitcoindev/CAPv7TjZTWhgzzdps3vb0Yo
> > U3EYJwThDFhNLkf4XmmdfhbORTaw%40mail.gmail.com
> >
> <https://groups.google.com/d/msgid/bitcoindev/CAPv7TjZTWhgzzdps3vb0Y
> >
> oU3EYJwThDFhNLkf4XmmdfhbORTaw%40mail.gmail.com?utm_medium=em
> > ail&utm_source=footer> .
> 


-- 
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups•com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/000201dbb7b7%247af02be0%2470d083a0%24%40voskuil.org.


 permalink raw reply	[|nested]
2025-04-27 16:45 [bitcoindev] The Tragic Tale of BIP30 Ruben Somsen
  2025-04-27 18:20 ` Luke Dashjr
  2025-04-27 18:30 ` eric

  2025-04-28 12:39   ` Eric Voskuil
  
From: Sjors Provoost @ 2025-04-28 11:48 UTC (permalink / raw)
  To: bitcoindev;  Ruben Somsen

Luke Dashr wrote:

> Solution C could be to remove it, but restore the previous UTXO. In other words, treat the overwrite as a spend of the overwritten transaction.

This might also be cleaner at the Bitcoin Core implementation level, because each UTXO set entry contains its creation height. And it might (conceptually) help Utreexo and SwiftSync.

Eric Voskuil wrote:

> I'm not aware of any compelling argument to hard fork out the existing checkpoints

Bitcoin Core plans to remove checkpoints entirely by v30 this fall. I just started a thread about this.

Ruben Somsen wrote:

> One last point to address is why BIP34 gets deactivated if block 227931 is reorged out. The reason for this is because otherwise it'd open the door to possibly creating outputs prior to BIP34's activation that will conflict with BIP34's rules for ensuring coinbase transaction uniqueness (the exact issue the Consensus Cleanup is seeking to resolve).
> 
> Ideally, it'd be nice to be able to sunset the BIP30 UTXO set check completely, ensuring it's no longer required, even in case of a reorg.

As I suggested in the other thread, it might be useful to have a more general BIP to describe the various issues around an alien-attack level reorg, for those who want to stick around and salvage the project.

In the case of BIP30, one option could be to have a rule that says: if the 2014 checkpoint is missing, then enforce the BIP54 Consensus Cleanup nLockTime rule from genesis. BIP34 can then simply go away.

(while you're at it, this rule could disable bare script, p2pk, p2sh, segwit v0, sigops counting and maybe some legacy stuff)

- Sjors

-- 
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups•com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/86FA0255-1A81-4DA1-9B1A-E57AD4F1DAAD%40sprovoost.nl.


 permalink raw reply	[|nested]
2025-04-28 11:48 ` Sjors Provoost

  2025-04-29 15:11     ` Ruben Somsen
  
From: Eric Voskuil @ 2025-04-28 12:39 UTC (permalink / raw)
  To: Sjors Provoost;  bitcoindev, Ruben Somsen


> On Apr 28, 2025, at 08:04, Sjors Provoost <sjors@sprovoost•nl> wrote:
> 
>> I'm not aware of any compelling argument to hard fork out the existing checkpoints
> 
> Bitcoin Core plans to remove checkpoints entirely by v30 this fall. I just started a thread about this.

The thread does not provide or reference any compelling argument (or any argument that I can find) for removing the checkpoints.

One reference states that “assume valid” speeds IBD, but of course it does so by not validating. So this does not provide a comparable offset to the increased validation cost, and even if it did does not provide an argument for hard forking them out.

e

-- 
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups•com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/E8225EAC-BED8-4840-8E3D-81A55C365209%40voskuil.org.


 permalink raw reply	[|nested]
2025-04-28 12:39   ` Eric Voskuil

  2025-04-29 15:28       ` Sjors Provoost
  2025-05-02 21:09       ` eric
  
From: Ruben Somsen @ 2025-04-29 15:11 UTC (permalink / raw)
  To: bitcoindev;  Eric Voskuil, Sjors Provoost, luke

[-- Attachment #1: Type: text/plain, Size: 2869 bytes --]

Hi,

Replying to parts of all messages thus far.


>The obvious solution to this problem is to not create the problem in the
first place.

Yes, that is a fair point. Not removing the checkpoints is one way of
ensuring the consensus bug cannot be triggered. I'm agnostic about whether
having checkpoints is also a reason to forgo consensus checks such as
BIP30 (or my proposed alternative of checking the coinbase TXID for
uniqueness and ensuring no future collision). Even though checkpoints
essentially force your node to halt if something were invalid up until that
point, I still think there is value in being able to verify that the rules
were followed.


>Solution C could be to remove it, but restore the previous UTXO

Yes, as Sjors also pointed out, I do think it is best to be precise about
which of the duplicates you're keeping. In fact, it's probably required to
ensure the rolling UTXO set hash remains consistent.


>In the case of BIP30, one option could be to have a rule that says: if the
2014 checkpoint is missing, then enforce the BIP54 Consensus Cleanup
nLockTime rule from genesis. BIP34 can then simply go away.

I'm afraid it's not that simple. If you wanted to fork off from some
arbitrary point prior to the last checkpoint, you'd want to enforce the new
consensus rules from that exact point (not from genesis), but that requires
shipping the full node software with a hash for every possible block that
could be forked off from. It's roughly 8MB of data so it's not impossible,
and I even had this written up as an alternative solution, but I removed it
in favor of the solution I ended up describing.


Cheers,
Ruben

On Mon, Apr 28, 2025 at 2:39 PM Eric Voskuil <eric@voskuil•org> wrote:

>
> > On Apr 28, 2025, at 08:04, Sjors Provoost <sjors@sprovoost•nl> wrote:
> >
> >> I'm not aware of any compelling argument to hard fork out the existing
> checkpoints
> >
> > Bitcoin Core plans to remove checkpoints entirely by v30 this fall. I
> just started a thread about this.
>
> The thread does not provide or reference any compelling argument (or any
> argument that I can find) for removing the checkpoints.
>
> One reference states that “assume valid” speeds IBD, but of course it does
> so by not validating. So this does not provide a comparable offset to the
> increased validation cost, and even if it did does not provide an argument
> for hard forking them out.
>
> e

-- 
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups•com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/CAPv7TjbEK8r8AQgzLvrwPmQHavU-ujgJqSz%2B7CFy_8W0_pwMvQ%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 3763 bytes --]

 permalink raw reply	[|nested]
2025-04-29 15:11     ` Ruben Somsen

  2025-05-02 21:09         ` eric
  2025-05-02 21:09       ` eric
  
From: Sjors Provoost @ 2025-04-29 15:28 UTC (permalink / raw)
  To: Ruben Somsen;  bitcoindev, Eric Voskuil, luke

Ruben Somsen wrote:

> >In the case of BIP30, one option could be to have a rule that says: if the 2014 checkpoint is missing, then enforce the BIP54 Consensus Cleanup nLockTime rule from genesis. BIP34 can then simply go away.
> 
> I'm afraid it's not that simple. If you wanted to fork off from some arbitrary point prior to the last checkpoint, you'd want to enforce the new consensus rules from that exact point (not from genesis), but that requires shipping the full node software with a hash for every possible block that could be forked off from. It's roughly 8MB of data so it's not impossible, and I even had this written up as an alternative solution, but I removed it in favor of the solution I ended up describing.

The trick is that no blocks obey the BIP54 rule for nLockTime, so they'll all be rejected and you can fork off starting at block 1.

- Sjors

-- 
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups•com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/ECC8258A-2DA5-4DD3-9D87-34D77C0C2C05%40sprovoost.nl.


 permalink raw reply	[|nested]
2025-04-29 15:28       ` Sjors Provoost

  2025-05-10 15:39           ` Eric Voskuil
  
From: eric @ 2025-05-02 21:09 UTC (permalink / raw)
  To: 'Sjors Provoost', 'Ruben Somsen';  bitcoindev, luke

Hi Sjors, 

> > >In the case of BIP30, one option could be to have a rule that says: if
the
> 2014 checkpoint is missing, then enforce the BIP54 Consensus Cleanup
> nLockTime rule from genesis. BIP34 can then simply go away.
> >
> > I'm afraid it's not that simple. If you wanted to fork off from some
arbitrary
> point prior to the last checkpoint, you'd want to enforce the new
consensus
> rules from that exact point (not from genesis), but that requires shipping
the
> full node software with a hash for every possible block that could be
forked off
> from. It's roughly 8MB of data so it's not impossible, and I even had this
> written up as an alternative solution, but I removed it in favor of the
solution I
> ended up describing.
> 
> The trick is that no blocks obey the BIP54 rule for nLockTime, so they'll
all be
> rejected and you can fork off starting at block 1.

This ignores the chain splits resulting from the 14 checkpoints that have
been removed to get to block 1. If the consensus is to not care about these
hard forks causing chain splits, there is really no reason to care about
this BIP30 chain split being caused by their removal.

Best,
Eric

-- 
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups•com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/035401dbbba6%247ea41790%247bec46b0%24%40voskuil.org.


 permalink raw reply	[|nested]
2025-04-29 15:11     ` Ruben Somsen
  2025-04-29 15:28       ` Sjors Provoost

  
From: eric @ 2025-05-02 21:09 UTC ( / )
  To: 'Ruben Somsen', bitcoindev;  'Sjors Provoost', luke

Hi Ruben,

> >The obvious solution to this problem is to not create the problem in the first
> place.
> 
> Yes, that is a fair point. Not removing the checkpoints is one way of ensuring
> the consensus bug cannot be triggered.

It's more than that. We are contemplating a "consensus bug" that would cause a chain split because the BIP30 exceptions are no longer being covered by any checkpoint. The heights at which this would cause a split are well below all but 3 of the 14 checkpoints. For that to occur 11 formerly checkpointed blocks would first have to be popped, given the existence of a stronger chain capable of triggering the above bug. This implies 11 more chain splits, depending at which point nodes adopted the checkpoint soft fork(s), just to reach this bug, and up to 14 possible in total. It makes no sense to fix this bug without first fixing chain splits that would be triggered *over 200,000 blocks less deep* than this BIP30 bug. And the only way to fix those is to not remove the checkpoints - which renders this bug inert.

> I'm agnostic about whether having
> checkpoints is also a reason to forgo consensus checks such as BIP30 (or my
> proposed alternative of checking the coinbase TXID for uniqueness and
> ensuring no future collision).

There is certainly a reason, the checkpoints are consensus rules.

> Even though checkpoints essentially force your node to halt if something were invalid up until that point, 

Right, if any other rule conflicted with them then the chain would stall forever at that point. That is not the applied meaning of these rules. The checkpoints declare that the blocks are required and therefore inherently valid. The reason we are having this conversation is the contemplated removal of the checkpoints, which also implies that other validation within covered blocks is not consensus.

> I still think there is value in being able to verify that the rules were followed.

Like them or not, checkpoints are the rules that are required to be followed.

> >Solution C could be to remove it, but restore the previous UTXO
> 
> Yes, as Sjors also pointed out, I do think it is best to be precise about which of
> the duplicates you're keeping. In fact, it's probably required to ensure the
> rolling UTXO set hash remains consistent.

Sure, but these are implementation details, not a matter of consensus. The consensus behavior would be that the second output is popped with its block and the first remains with its block. Assuming it would be written up, that's how I would recommend handling it.

Best,
Eric

-- 
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups•com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/035301dbbba6%24701dd490%2450597db0%24%40voskuil.org.
2025-05-02 21:09         ` eric

  2025-05-10 16:17             ` Sjors Provoost
  
From: Eric Voskuil @ 2025-05-10 15:39 UTC (permalink / raw)
  To: Sjors Provoost, Ruben Somsen;  bitcoindev, luke

This thread seems to have gone silent. Are these pending hard forks no longer interesting?

e

> This ignores the chain splits resulting from the 14 checkpoints that have
> been removed to get to block 1. If the consensus is to not care about these
> hard forks causing chain splits, there is really no reason to care about
> this BIP30 chain split being caused by their removal.
> 
> Best,
> Eric

-- 
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups•com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/ECE5766B-F902-4CA4-9DFC-7718C1DA4173%40voskuil.org.


 permalink raw reply	[|nested]
2025-05-10 15:39           ` Eric Voskuil

  2025-05-10 16:55               ` Eric Voskuil
  
From: Sjors Provoost @ 2025-05-10 16:17 UTC (permalink / raw)
  To: bitcoindev

Hi Eric,

I agree that deep-reorg BIP30 handling is not important. Although it _is_ an interesting exercise which helps to better understand consensus code. I think people got distracted a bit by recent drama. 

The Bitcoin Core project "decided" many years ago to not prioritise the graceful handling of extremely deep reorgs. You already stated your disagreement with that approach back then. The dropping of checkpoints is a continuation of that.

The only thing that would motivate me to bring back checkpoints (i.e. undo the PR that dropped them) is an attack that doesn't involve alien technology.

At the same time I don't object to, and might even review, changes that:

1. are simple enough, like Solution C earlier in the thread; or
2. someone writes a thorough BIP that goes though all the ways different (versions of) implementations handle extreme reorgs, and comes up with simple mitigations that make the handling consistent

As long as they don't bring checkpoints back. I think they've outlived their usefulness as consensus training wheels and now they're just an invitation for legal attacks (or future developer laziness).

- Sjors

> Op 10 mei 2025, om 17:39 heeft Eric Voskuil <eric@voskuil•org> het volgende geschreven:
> 
> This thread seems to have gone silent. Are these pending hard forks no longer interesting?
> 
> e
> 
>> This ignores the chain splits resulting from the 14 checkpoints that have
>> been removed to get to block 1. If the consensus is to not care about these
>> hard forks causing chain splits, there is really no reason to care about
>> this BIP30 chain split being caused by their removal.
>> 
>> Best,
>> Eric

-- 
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups•com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/4AC2B1A6-23F3-4A06-808F-448D9DD58FE2%40sprovoost.nl.


 permalink raw reply	[|nested]
2025-05-10 16:17             ` Sjors Provoost

  
From: Eric Voskuil @ 2025-05-10 16:55 UTC (permalink / raw)
  To: Sjors Provoost;  bitcoindev

Hi Sjors,

The point I’m making is that introducing more complexity to try and mitigate this issue, while ignoring far more likely issues resulting from these various hard forks, seems irrational and inconsistent. So either way I would close out this question.

It would make more sense to me to have the project(s) that have deployed such hard forks finally BIP all of them, as some have been deployed silently. This would allow all implementations to know what the consensus might actually be (or not) without having to fish through source code repos.

At least one of these supposedly inert hard forks (silently deployed) has resulted in a very likely future chain split, which the Consensus Cleanup effort is attempting to mitigate. Interestingly enough it pertains directly to bip30.

At least this way various projects and their consumers can make informed decisions about how they perceive the risk and the benefit. Some might simply take BC at its word (that removing checkpoints is unlikely to cause a chain split) and simply ignore it.

e

> On May 10, 2025, at 12:24, Sjors Provoost <sjors@sprovoost•nl> wrote:
> 
> ﻿Hi Eric,
> 
> I agree that deep-reorg BIP30 handling is not important. Although it _is_ an interesting exercise which helps to better understand consensus code. I think people got distracted a bit by recent drama.
> 
> The Bitcoin Core project "decided" many years ago to not prioritise the graceful handling of extremely deep reorgs. You already stated your disagreement with that approach back then. The dropping of checkpoints is a continuation of that.
> 
> The only thing that would motivate me to bring back checkpoints (i.e. undo the PR that dropped them) is an attack that doesn't involve alien technology.
> 
> At the same time I don't object to, and might even review, changes that:
> 
> 1. are simple enough, like Solution C earlier in the thread; or
> 2. someone writes a thorough BIP that goes though all the ways different (versions of) implementations handle extreme reorgs, and comes up with simple mitigations that make the handling consistent
> 
> As long as they don't bring checkpoints back. I think they've outlived their usefulness as consensus training wheels and now they're just an invitation for legal attacks (or future developer laziness).
> 
> - Sjors
> 
>> Op 10 mei 2025, om 17:39 heeft Eric Voskuil <eric@voskuil•org> het volgende geschreven:
>> 
>> This thread seems to have gone silent. Are these pending hard forks no longer interesting?
>> 
>> e
>> 
>>> This ignores the chain splits resulting from the 14 checkpoints that have
>>> been removed to get to block 1. If the consensus is to not care about these
>>> hard forks causing chain splits, there is really no reason to care about
>>> this BIP30 chain split being caused by their removal.
>>> 
>>> Best,
>>> Eric
> 
> --
> You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups•com.
> To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/4AC2B1A6-23F3-4A06-808F-448D9DD58FE2%40sprovoost.nl.

-- 
You received this message because you are subscribed to the Google Groups "Bitcoin Development Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bitcoindev+unsubscribe@googlegroups•com.
To view this discussion visit https://groups.google.com/d/msgid/bitcoindev/197BF722-4D8F-4796-8FBC-1F002D2CCE31%40voskuil.org.


 permalink raw reply	[|nested]