Reasons to add sync flags to Bitcoin

Reasons to add sync flags to Bitcoin
2016-07-26 13:51 ` Tom
                   ` (2 more replies)
  
From: Moral Agent @ 2016-07-26 12:47 UTC (permalink / raw)
  To: bitcoin-dev

[-- Attachment #1: Type: text/plain, Size: 8353 bytes --]

I posted this to /r/bitcoin yesterday but it got minimal comments. One uses
suggested I try the mailing list so here it is:

The idea presented here could have the following benefits:

1. Improve mining decentralization
2. Reduce variance in mining profitability
3. Reduce or eliminate SPV mined blocks
4. Reduce or eliminate empty blocks, smoothing out resource usage
5. Reduce or eliminate the latency bottleneck on throughput
6. Make transaction stuffing by miners be either obvious or costly
7. Gives miners something to do while they wait for attractive transactions
to appear
8. Can be easily done with a soft fork

#Basic idea:

Ideally, all miners would begin hashing the next block at exactly the same
time. Miners with a head start are more profitable, and the techniques that
help miners receive and validate blocks quickly create centralization
pressure.

What if there was something that acted like the starting flag at a race,
which could suddenly wave and cause all of the miners to simultaneously
begin hashing the next block?

#Implementation:

Let a sync flag be a message consisting of:

1. Hash of the previous block.
2. Bitcoin address
3. Nonce

This tiny message could propagate through the network at maximum speed. If
miners had to include the hash of this flag in the next block, then all
miners wait for this flag, and when it suddenly spread through the network,
all miners could simultaneously begin hashing the next block.

The sync flag should not be produced too quickly. You want to give everyone
enough time to be ready to hash the next block. Let's say that the hash of
the sync flag is a proof of work that is targeted for 2 minutes.

To fund this proof of work, the protocol is modified to demand that the
block produced 10 blocks after the sync flag must allocate 25% of the block
reward to the address published by the sync flag. In this way, sync flags
are produced in 2 minutes, and blocks are produced in 8 minutes, with 10
minutes total.


Illustration 1: https://s32.postimg.org/wzg0hs8lx/sync_flag.png)

Illustration 2: https://s32.postimg.org/vc5y9yz4l/sync_flag2.png


#Explanation of reasons:

**Improve mining decentralization**

One factor driving centralization is the imperative miners have to achieve
low latency in receiving and validating blocks. To achieve low latency, it
helps a lot if you have expensive low-latency internet connections, curated
network topologies, and large pools that have a plausible chance of finding
consecutive blocks. If miners are expected (or forced) to validate a block
prior to mining on top of it, the rational end game would be to outsource
the validation step to a trusted third party specialist who can choose
optimal locations on the globe to serve their (multiple?) mining pool
clients. These are all less decentralized than the mining situation Satoshi
and others imagined.

**Reduce variance in mining revenue**

Currently, there are about 144 opportunities per day for a pool or solo
miner to see any revenue at all. With sync flags, that number doubles to
288. Sync flags are only worth 25% of what a block is worth, but this still
represents a significant reduction in variance. This variance is one factor
causing solo miners to group into pools, and large pools to be more
attractive than small pools.

**Reduce or eliminate SPV mined blocks**

One way miners have sought to make
full-block-transmission-and-validation-latency irrelevant has been through
"SPV" mining or "Head-first" mining. There is some evidence that these
techniques may be widely used, and that badgering the miners about it is an
ineffective strategy to stop them.

In SPV mining, a miner would simply accept any block header that shows the
correct proof of work. All other validation is entrusted to other miners.
This practice is quite dangerous as the SPV miners can wander off on some
invalid chain, taking SPV nodes with them. If this occurs during a soft
fork, these blind miners can also fool unupgraded fully validating nodes
into following them.

"Head-first" mining means that miners start hashing as soon as they receive
the block header with the correct POW, but they simultaneously validate the
block, and abandon it if is not valid. I consider this to be pretty safe,
as it strictly limits the length of an invalid chain that can result from
mining without validating. However, "Head-first" mining can plausibly
generate 2 or 3 confirmations of an invalid block. It would be nice if such
confirmations did not happen.

The sync flag technique is similar to head-first mining, but rather than
hashing the next block while they wait for transmission and validation of
the prior block, they hash the sync flag. Nodes can differentiate between
sync flags and blocks, and can ignore sync flags when counting
confirmations.

**Reduce or eliminate empty blocks, smoothing out resource usage**

Empty blocks are another consequence of SPV or Headfirst mining, because
the miner cannot safely include any transactions in the block they are
hashing until they have validated the prior block. By delaying the start of
hashing the next block until after validation, miners would not have this
reason to mine empty blocks.

**Reduce or eliminate the latency bottleneck on throughput**

Centralization pressure due to latency issues has been a major
preoccupation over the last year. If latency mattered much less, it could
represent a scalability improvement that could enable higher throughput.

**Make transaction stuffing by miners be either obvious or costly**

Currently, the entire block reward goes to the miner who mines it. One
unfortunate consequence of this is that it does not cost the miner anything
to covertly stuff the block with transactions. These transactions would pay
fees and be indistinguishable from ordinary transactions, but the fees are
paid by the miner and then immediately returned to the miner.

With sync flags, the miner must share these transaction fees with the
address contained in the sync flag 10 blocks prior. This means that if the
miner gives the transactions a normal looking fee, they will incur a cost
that will be paid to the sync flag. If the miner wants to avoid this, they
must give their stuffing transactions a zero fee, which provides evidence
that they are stuffing.

Also, when miners stuff with transactions using a zero fee, they cannot
manipulate the perception of how much fee it takes to get into a block.

Note that miners could still try to covertly stuff blocks that will pay a
sync flag that they themselves created. if this is a big concern, it can be
addressed by forcing blocks to pay multiple sync flags.

**Gives miners something to do while they wait for attractive transactions
to appear**

From the Montreal scaling workshop last year, we have [this talk](
https://scalingbitcoin.org/montreal2015/presentations/Day1/13-miles-carlsten-Mind-the-Gap.pdf)
which worried that as the block subsidy reduced and transactions became a
more important fraction of miner revenue, it would be rational for miners
to turn off their mining equipment for a "gap" phase after a block is
found, to allow time to pass as more lucrative transactions entered the
mempool.

I don't know whether this will actually happen. The presence of a suitable
backlog of transactions would help prevent this dynamic from emerging. But
if such idling behavior was the optima mining strategy, it could create a
serious vulnerability. Idle hands are the devil's workshop as the saying
goes, and idle miners represent a pool of inert hashpower that is available
to rent for all kinds of destabilizing purposes. It would be better to put
those miners to profitable work mining a sync flag while they wait.

Also, this creates a more efficient price discovery mechanism for
transactions, because you allow transactions paying high fees time to
arrive to the marketplace, rather than take whatever anyone is offering
because all the "good" transactions got gobbled up in the prior block.

**Can be easily done with a soft fork**

Although a hard fork would be more efficient, sync flags could be easily
implemented using a soft fork by introducing the following rule:

Every block must include a transaction which pays 25% of the block reward
to the address given by the 10th previous sync flag, and commits to the
hash of the 1st previous sync flag.

[-- Attachment #2: Type: text/html, Size: 9745 bytes --]

 permalink raw reply	[|nested]
2016-07-26 13:51 ` 
                   ` 
  
From: Moral Agent @ 2016-07-26 12:47 UTC (permalink / raw)
  To: bitcoin-dev

[-- Attachment #1: Type: text/plain, Size: 8353 bytes --]

I posted this to /r/bitcoin yesterday but it got minimal comments. One uses
suggested I try the mailing list so here it is:

The idea presented here could have the following benefits:

1. Improve mining decentralization
2. Reduce variance in mining profitability
3. Reduce or eliminate SPV mined blocks
4. Reduce or eliminate empty blocks, smoothing out resource usage
5. Reduce or eliminate the latency bottleneck on throughput
6. Make transaction stuffing by miners be either obvious or costly
7. Gives miners something to do while they wait for attractive transactions
to appear
8. Can be easily done with a soft fork

#Basic idea:

Ideally, all miners would begin hashing the next block at exactly the same
time. Miners with a head start are more profitable, and the techniques that
help miners receive and validate blocks quickly create centralization
pressure.

What if there was something that acted like the starting flag at a race,
which could suddenly wave and cause all of the miners to simultaneously
begin hashing the next block?

#Implementation:

Let a sync flag be a message consisting of:

1. Hash of the previous block.
2. Bitcoin address
3. Nonce

This tiny message could propagate through the network at maximum speed. If
miners had to include the hash of this flag in the next block, then all
miners wait for this flag, and when it suddenly spread through the network,
all miners could simultaneously begin hashing the next block.

The sync flag should not be produced too quickly. You want to give everyone
enough time to be ready to hash the next block. Let's say that the hash of
the sync flag is a proof of work that is targeted for 2 minutes.

To fund this proof of work, the protocol is modified to demand that the
block produced 10 blocks after the sync flag must allocate 25% of the block
reward to the address published by the sync flag. In this way, sync flags
are produced in 2 minutes, and blocks are produced in 8 minutes, with 10
minutes total.


Illustration 1: https://s32.postimg.org/wzg0hs8lx/sync_flag.png)

Illustration 2: https://s32.postimg.org/vc5y9yz4l/sync_flag2.png


#Explanation of reasons:

**Improve mining decentralization**

One factor driving centralization is the imperative miners have to achieve
low latency in receiving and validating blocks. To achieve low latency, it
helps a lot if you have expensive low-latency internet connections, curated
network topologies, and large pools that have a plausible chance of finding
consecutive blocks. If miners are expected (or forced) to validate a block
prior to mining on top of it, the rational end game would be to outsource
the validation step to a trusted third party specialist who can choose
optimal locations on the globe to serve their (multiple?) mining pool
clients. These are all less decentralized than the mining situation Satoshi
and others imagined.

**Reduce variance in mining revenue**

Currently, there are about 144 opportunities per day for a pool or solo
miner to see any revenue at all. With sync flags, that number doubles to
288. Sync flags are only worth 25% of what a block is worth, but this still
represents a significant reduction in variance. This variance is one factor
causing solo miners to group into pools, and large pools to be more
attractive than small pools.

**Reduce or eliminate SPV mined blocks**

One way miners have sought to make
full-block-transmission-and-validation-latency irrelevant has been through
"SPV" mining or "Head-first" mining. There is some evidence that these
techniques may be widely used, and that badgering the miners about it is an
ineffective strategy to stop them.

In SPV mining, a miner would simply accept any block header that shows the
correct proof of work. All other validation is entrusted to other miners.
This practice is quite dangerous as the SPV miners can wander off on some
invalid chain, taking SPV nodes with them. If this occurs during a soft
fork, these blind miners can also fool unupgraded fully validating nodes
into following them.

"Head-first" mining means that miners start hashing as soon as they receive
the block header with the correct POW, but they simultaneously validate the
block, and abandon it if is not valid. I consider this to be pretty safe,
as it strictly limits the length of an invalid chain that can result from
mining without validating. However, "Head-first" mining can plausibly
generate 2 or 3 confirmations of an invalid block. It would be nice if such
confirmations did not happen.

The sync flag technique is similar to head-first mining, but rather than
hashing the next block while they wait for transmission and validation of
the prior block, they hash the sync flag. Nodes can differentiate between
sync flags and blocks, and can ignore sync flags when counting
confirmations.

**Reduce or eliminate empty blocks, smoothing out resource usage**

Empty blocks are another consequence of SPV or Headfirst mining, because
the miner cannot safely include any transactions in the block they are
hashing until they have validated the prior block. By delaying the start of
hashing the next block until after validation, miners would not have this
reason to mine empty blocks.

**Reduce or eliminate the latency bottleneck on throughput**

Centralization pressure due to latency issues has been a major
preoccupation over the last year. If latency mattered much less, it could
represent a scalability improvement that could enable higher throughput.

**Make transaction stuffing by miners be either obvious or costly**

Currently, the entire block reward goes to the miner who mines it. One
unfortunate consequence of this is that it does not cost the miner anything
to covertly stuff the block with transactions. These transactions would pay
fees and be indistinguishable from ordinary transactions, but the fees are
paid by the miner and then immediately returned to the miner.

With sync flags, the miner must share these transaction fees with the
address contained in the sync flag 10 blocks prior. This means that if the
miner gives the transactions a normal looking fee, they will incur a cost
that will be paid to the sync flag. If the miner wants to avoid this, they
must give their stuffing transactions a zero fee, which provides evidence
that they are stuffing.

Also, when miners stuff with transactions using a zero fee, they cannot
manipulate the perception of how much fee it takes to get into a block.

Note that miners could still try to covertly stuff blocks that will pay a
sync flag that they themselves created. if this is a big concern, it can be
addressed by forcing blocks to pay multiple sync flags.

**Gives miners something to do while they wait for attractive transactions
to appear**

From the Montreal scaling workshop last year, we have [this talk](
https://scalingbitcoin.org/montreal2015/presentations/Day1/13-miles-carlsten-Mind-the-Gap.pdf)
which worried that as the block subsidy reduced and transactions became a
more important fraction of miner revenue, it would be rational for miners
to turn off their mining equipment for a "gap" phase after a block is
found, to allow time to pass as more lucrative transactions entered the
mempool.

I don't know whether this will actually happen. The presence of a suitable
backlog of transactions would help prevent this dynamic from emerging. But
if such idling behavior was the optima mining strategy, it could create a
serious vulnerability. Idle hands are the devil's workshop as the saying
goes, and idle miners represent a pool of inert hashpower that is available
to rent for all kinds of destabilizing purposes. It would be better to put
those miners to profitable work mining a sync flag while they wait.

Also, this creates a more efficient price discovery mechanism for
transactions, because you allow transactions paying high fees time to
arrive to the marketplace, rather than take whatever anyone is offering
because all the "good" transactions got gobbled up in the prior block.

**Can be easily done with a soft fork**

Although a hard fork would be more efficient, sync flags could be easily
implemented using a soft fork by introducing the following rule:

Every block must include a transaction which pays 25% of the block reward
to the address given by the 10th previous sync flag, and commits to the
hash of the 1st previous sync flag.

[-- Attachment #2: Type: text/html, Size: 9745 bytes --]

 permalink raw reply	[|nested]
2016-07-26 13:51 ` Tom

  
From: Erik Aronesty @ 2016-07-26 17:27 UTC (permalink / raw)
  To: Tom, Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 2178 bytes --]

   - Flags will be mined selfishly, and not published until the advantage
   gained from withholding is less than the mining reward.  This effect may
   kill the decentralization features, since big miners will be the only ones
   that can selfish-mine flags.  Indeed, collusion would be encouraged... just
   ship the flag to the miners you do business with, and no one else.   At the
   expense of loss of flag revenue, your in-group would gain a massive
   advantage in main-chain mining.


On Tue, Jul 26, 2016 at 9:51 AM, Tom via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> wrote:

> > #Basic idea:
> >
> > Ideally, all miners would begin hashing the next block at exactly the
> same
> > time. Miners with a head start are more profitable, and the techniques
> that
> > help miners receive and validate blocks quickly create centralization
> > pressure.
> >
> > What if there was something that acted like the starting flag at a race,
> > which could suddenly wave and cause all of the miners to simultaneously
> > begin hashing the next block?
> >
> > #Implementation:
> >
> > Let a sync flag be a message consisting of:
> >
> > 1. Hash of the previous block.
> > 2. Bitcoin address
> > 3. Nonce
> >
> > This tiny message could propagate through the network at maximum speed.
> If
> > miners had to include the hash of this flag in the next block, then all
> > miners wait for this flag, and when it suddenly spread through the
> network,
> > all miners could simultaneously begin hashing the next block.
>
> What you describe in this part of your message can be done with no forks
> whatsoever and I think that this is enough. Don't really see the reason for
> any change in funding.
>
> The idea of sending out a block header is essentially what I called
> "optimistic mining" and has been described in more detail in my blog here;
> http://zander.github.io/posts/Innovation%20-%20OnlineScaling/
>
> The video explains with graphics too...
>
> You may find this interesting :)
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 3007 bytes --]

 permalink raw reply	[|nested]
2016-07-26 21:45 ` Tier Nolan
  
From: Martijn Meijering @ 2016-07-26 20:58 UTC (permalink / raw)
  To: bitcoin-dev

- Flags will be mined selfishly, and not published until the advantage
   gained from withholding is less than the mining reward.  This effect may
   kill the decentralization features, since big miners will be the only ones
   that can selfish-mine flags.  Indeed, collusion would be encouraged... just
   ship the flag to the miners you do business with, and no one else.   At the
   expense of loss of flag revenue, your in-group would gain a massive
   advantage in main-chain mining.

---

Is there a reason miners would be more likely to engage in selfish
mining of sync flags than they are now with ordinary blocks?


 permalink raw reply	[|nested]
2016-07-26 20:58 Martijn Meijering

  
From: Tier Nolan @ 2016-07-26 21:45 UTC (permalink / raw)
  To: Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 1202 bytes --]

On Tue, Jul 26, 2016 at 9:58 PM, Martijn Meijering via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> wrote:

> Is there a reason miners would be more likely to engage in selfish
> mining of sync flags than they are now with ordinary blocks?
>


This proposal has the same effect as adding mandatory empty blocks.

POW targeted at 2 minutes means that the POW for the flag is 25% of the
block POW.  That gives a flag every 2 minutes and a block every 8 minutes.

It has the feature that the conversion rate from hashing power to reward is
the same for the flags and the blocks.  A flag get 25% of the reward for
25% of the effort.

A soft fork to add this rule would have a disadvantage relative to a
competing chain.  It would divert 20% of its hashing power to the flag
blocks, which would be ignored by legacy nodes.  The soft fork would need
55% of the hashing power to win the race.

This isn't that big a deal if a 75% activation threshold is used.  It might
be worth bumping it up to 80% in that case.

This rule would mean that headers first clients would have to download more
information to verify the longest chain.  If they only download the
headers, they are missing 20% of the POW.

[-- Attachment #2: Type: text/html, Size: 1639 bytes --]

 permalink raw reply	[|nested]
2016-07-26 12:47 [bitcoin-dev] Reasons to add sync flags to Bitcoin Moral Agent
  2016-07-26 13:51 ` Tom

  2016-07-27 14:42   ` Moral Agent
  2016-07-28 16:41 ` Moral Agent
  
From: Nick ODell @ 2016-07-26 22:03 UTC (permalink / raw)
  To: Moral Agent, Bitcoin Protocol Discussion

Moral,

Mining the sync flag isn't compatible with the payout structure of non
hot-wallet pools like Eligius or decentralized pools like p2pool.
Those need the ability to split a reward among multiple parties.
Instead of giving an address to send the funds to, you could include
the hash of the transaction allowed to spend the sync flag output.
You'd have to zero the previous outpoint of the transaction before
hashing, since you don't know what the hash of the coinbase ten blocks
from now will be.


On Tue, Jul 26, 2016 at 6:47 AM, Moral Agent via bitcoin-dev
<bitcoin-dev@lists•linuxfoundation.org> wrote:
> I posted this to /r/bitcoin yesterday but it got minimal comments. One uses
> suggested I try the mailing list so here it is:
>
> The idea presented here could have the following benefits:
>
> 1. Improve mining decentralization
> 2. Reduce variance in mining profitability
> 3. Reduce or eliminate SPV mined blocks
> 4. Reduce or eliminate empty blocks, smoothing out resource usage
> 5. Reduce or eliminate the latency bottleneck on throughput
> 6. Make transaction stuffing by miners be either obvious or costly
> 7. Gives miners something to do while they wait for attractive transactions
> to appear
> 8. Can be easily done with a soft fork
>
> #Basic idea:
>
> Ideally, all miners would begin hashing the next block at exactly the same
> time. Miners with a head start are more profitable, and the techniques that
> help miners receive and validate blocks quickly create centralization
> pressure.
>
> What if there was something that acted like the starting flag at a race,
> which could suddenly wave and cause all of the miners to simultaneously
> begin hashing the next block?
>
> #Implementation:
>
> Let a sync flag be a message consisting of:
>
> 1. Hash of the previous block.
> 2. Bitcoin address
> 3. Nonce
>
> This tiny message could propagate through the network at maximum speed. If
> miners had to include the hash of this flag in the next block, then all
> miners wait for this flag, and when it suddenly spread through the network,
> all miners could simultaneously begin hashing the next block.
>
> The sync flag should not be produced too quickly. You want to give everyone
> enough time to be ready to hash the next block. Let's say that the hash of
> the sync flag is a proof of work that is targeted for 2 minutes.
>
> To fund this proof of work, the protocol is modified to demand that the
> block produced 10 blocks after the sync flag must allocate 25% of the block
> reward to the address published by the sync flag. In this way, sync flags
> are produced in 2 minutes, and blocks are produced in 8 minutes, with 10
> minutes total.
>
>
> Illustration 1: https://s32.postimg.org/wzg0hs8lx/sync_flag.png)
>
> Illustration 2: https://s32.postimg.org/vc5y9yz4l/sync_flag2.png
>
>
> #Explanation of reasons:
>
> **Improve mining decentralization**
>
> One factor driving centralization is the imperative miners have to achieve
> low latency in receiving and validating blocks. To achieve low latency, it
> helps a lot if you have expensive low-latency internet connections, curated
> network topologies, and large pools that have a plausible chance of finding
> consecutive blocks. If miners are expected (or forced) to validate a block
> prior to mining on top of it, the rational end game would be to outsource
> the validation step to a trusted third party specialist who can choose
> optimal locations on the globe to serve their (multiple?) mining pool
> clients. These are all less decentralized than the mining situation Satoshi
> and others imagined.
>
> **Reduce variance in mining revenue**
>
> Currently, there are about 144 opportunities per day for a pool or solo
> miner to see any revenue at all. With sync flags, that number doubles to
> 288. Sync flags are only worth 25% of what a block is worth, but this still
> represents a significant reduction in variance. This variance is one factor
> causing solo miners to group into pools, and large pools to be more
> attractive than small pools.
>
> **Reduce or eliminate SPV mined blocks**
>
> One way miners have sought to make
> full-block-transmission-and-validation-latency irrelevant has been through
> "SPV" mining or "Head-first" mining. There is some evidence that these
> techniques may be widely used, and that badgering the miners about it is an
> ineffective strategy to stop them.
>
> In SPV mining, a miner would simply accept any block header that shows the
> correct proof of work. All other validation is entrusted to other miners.
> This practice is quite dangerous as the SPV miners can wander off on some
> invalid chain, taking SPV nodes with them. If this occurs during a soft
> fork, these blind miners can also fool unupgraded fully validating nodes
> into following them.
>
> "Head-first" mining means that miners start hashing as soon as they receive
> the block header with the correct POW, but they simultaneously validate the
> block, and abandon it if is not valid. I consider this to be pretty safe, as
> it strictly limits the length of an invalid chain that can result from
> mining without validating. However, "Head-first" mining can plausibly
> generate 2 or 3 confirmations of an invalid block. It would be nice if such
> confirmations did not happen.
>
> The sync flag technique is similar to head-first mining, but rather than
> hashing the next block while they wait for transmission and validation of
> the prior block, they hash the sync flag. Nodes can differentiate between
> sync flags and blocks, and can ignore sync flags when counting
> confirmations.
>
> **Reduce or eliminate empty blocks, smoothing out resource usage**
>
> Empty blocks are another consequence of SPV or Headfirst mining, because the
> miner cannot safely include any transactions in the block they are hashing
> until they have validated the prior block. By delaying the start of hashing
> the next block until after validation, miners would not have this reason to
> mine empty blocks.
>
> **Reduce or eliminate the latency bottleneck on throughput**
>
> Centralization pressure due to latency issues has been a major preoccupation
> over the last year. If latency mattered much less, it could represent a
> scalability improvement that could enable higher throughput.
>
> **Make transaction stuffing by miners be either obvious or costly**
>
> Currently, the entire block reward goes to the miner who mines it. One
> unfortunate consequence of this is that it does not cost the miner anything
> to covertly stuff the block with transactions. These transactions would pay
> fees and be indistinguishable from ordinary transactions, but the fees are
> paid by the miner and then immediately returned to the miner.
>
> With sync flags, the miner must share these transaction fees with the
> address contained in the sync flag 10 blocks prior. This means that if the
> miner gives the transactions a normal looking fee, they will incur a cost
> that will be paid to the sync flag. If the miner wants to avoid this, they
> must give their stuffing transactions a zero fee, which provides evidence
> that they are stuffing.
>
> Also, when miners stuff with transactions using a zero fee, they cannot
> manipulate the perception of how much fee it takes to get into a block.
>
> Note that miners could still try to covertly stuff blocks that will pay a
> sync flag that they themselves created. if this is a big concern, it can be
> addressed by forcing blocks to pay multiple sync flags.
>
> **Gives miners something to do while they wait for attractive transactions
> to appear**
>
> From the Montreal scaling workshop last year, we have [this
> talk](https://scalingbitcoin.org/montreal2015/presentations/Day1/13-miles-carlsten-Mind-the-Gap.pdf)
> which worried that as the block subsidy reduced and transactions became a
> more important fraction of miner revenue, it would be rational for miners to
> turn off their mining equipment for a "gap" phase after a block is found, to
> allow time to pass as more lucrative transactions entered the mempool.
>
> I don't know whether this will actually happen. The presence of a suitable
> backlog of transactions would help prevent this dynamic from emerging. But
> if such idling behavior was the optima mining strategy, it could create a
> serious vulnerability. Idle hands are the devil's workshop as the saying
> goes, and idle miners represent a pool of inert hashpower that is available
> to rent for all kinds of destabilizing purposes. It would be better to put
> those miners to profitable work mining a sync flag while they wait.
>
> Also, this creates a more efficient price discovery mechanism for
> transactions, because you allow transactions paying high fees time to arrive
> to the marketplace, rather than take whatever anyone is offering because all
> the "good" transactions got gobbled up in the prior block.
>
> **Can be easily done with a soft fork**
>
> Although a hard fork would be more efficient, sync flags could be easily
> implemented using a soft fork by introducing the following rule:
>
> Every block must include a transaction which pays 25% of the block reward to
> the address given by the 10th previous sync flag, and commits to the hash of
> the 1st previous sync flag.
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>


 permalink raw reply	[|nested]
2016-07-26 22:03 ` Nick ODell

  
From: Moral Agent @ 2016-07-27 14:42 UTC (permalink / raw)
  To: bitcoin-dev

[-- Attachment #1: Type: text/plain, Size: 12417 bytes --]

I made a repo to be a home for sync_flags here:
https://github.com/moral-agent/sync_flags

If you see any personally identifying information, please be a good sport
and let me know. I'm a nobody, but I'd still prefer not to get doxxed.

Two changes to the proposal (see repo for explanations)

1. Sync flags now would have the same difficulty as blocks.
2. Blocks now donate to 5 sync flags instead of 1

I also added comments about selfish mining and invalid block spam.

Response to replies:

tomz@freedommail•ch: What is the advantage over optimistic mining?

1. Sync flags can be somewhat smaller than block headers.
2. Sync flags improve variance by doubling the number of chances to win
3. Sync flags can be distinguished from normal blocks, so SPV clients can
ignore them as confirmations.
4. Sync flags reward all miners equally. Optimistic blocks have to be empty
unless you mined the previous block, which damages decentralization.
5. Sync flags result in fewer empty blocks, smoothing out resource usage
6. Sync flags make transaction stuffing by miners either obvious or costly
7. Sync flags give miners something to do while they wait for attractive
transactions to appear.

erik@q32•com: Flags will be selfish mined.

I agree that flags would likely be selfish mined. I have modified the
proposal to say that flags have the same POW target as blocks, so the
selfish mining vulnerability should be equal to the current protocol.

martijn.meijering@mevs•nl: Why expect more selfish mining?

Because flags had small POW relative to blocks. After you find a block, why
not hide it while you take a crack at the flag?

tier.nolan@gmail•com: Effect is same as mandatory empty blocks.

Not quite the same:

1. Sync flags can be somewhat smaller than block headers.
2. Sync flags can be distinguished from normal blocks, so SPV clients can
ignore them as confirmations.
3. Sync flags make transaction stuffing by miners either obvious or costly
4. No one pays for empty blocks, except for the block subsidy. Some miners
may choose to only mine the non-empty blocks, resulting in
hashpower-for-rent to make mischief or hashpower that oscillates, creating
a situation where empty blocks take longer to mine than full blocks.

nickodell@gmail•com: Payout mechanism incompatible with certain mining pools

Hopefully some kind of smart contract structure could be implemented as you
suggested.


On Tue, Jul 26, 2016 at 6:03 PM, Nick ODell <nickodell@gmail•com> wrote:

> Moral,
>
> Mining the sync flag isn't compatible with the payout structure of non
> hot-wallet pools like Eligius or decentralized pools like p2pool.
> Those need the ability to split a reward among multiple parties.
> Instead of giving an address to send the funds to, you could include
> the hash of the transaction allowed to spend the sync flag output.
> You'd have to zero the previous outpoint of the transaction before
> hashing, since you don't know what the hash of the coinbase ten blocks
> from now will be.
>
>
> On Tue, Jul 26, 2016 at 6:47 AM, Moral Agent via bitcoin-dev
> <bitcoin-dev@lists•linuxfoundation.org> wrote:
> > I posted this to /r/bitcoin yesterday but it got minimal comments. One
> uses
> > suggested I try the mailing list so here it is:
> >
> > The idea presented here could have the following benefits:
> >
> > 1. Improve mining decentralization
> > 2. Reduce variance in mining profitability
> > 3. Reduce or eliminate SPV mined blocks
> > 4. Reduce or eliminate empty blocks, smoothing out resource usage
> > 5. Reduce or eliminate the latency bottleneck on throughput
> > 6. Make transaction stuffing by miners be either obvious or costly
> > 7. Gives miners something to do while they wait for attractive
> transactions
> > to appear
> > 8. Can be easily done with a soft fork
> >
> > #Basic idea:
> >
> > Ideally, all miners would begin hashing the next block at exactly the
> same
> > time. Miners with a head start are more profitable, and the techniques
> that
> > help miners receive and validate blocks quickly create centralization
> > pressure.
> >
> > What if there was something that acted like the starting flag at a race,
> > which could suddenly wave and cause all of the miners to simultaneously
> > begin hashing the next block?
> >
> > #Implementation:
> >
> > Let a sync flag be a message consisting of:
> >
> > 1. Hash of the previous block.
> > 2. Bitcoin address
> > 3. Nonce
> >
> > This tiny message could propagate through the network at maximum speed.
> If
> > miners had to include the hash of this flag in the next block, then all
> > miners wait for this flag, and when it suddenly spread through the
> network,
> > all miners could simultaneously begin hashing the next block.
> >
> > The sync flag should not be produced too quickly. You want to give
> everyone
> > enough time to be ready to hash the next block. Let's say that the hash
> of
> > the sync flag is a proof of work that is targeted for 2 minutes.
> >
> > To fund this proof of work, the protocol is modified to demand that the
> > block produced 10 blocks after the sync flag must allocate 25% of the
> block
> > reward to the address published by the sync flag. In this way, sync flags
> > are produced in 2 minutes, and blocks are produced in 8 minutes, with 10
> > minutes total.
> >
> >
> > Illustration 1: https://s32.postimg.org/wzg0hs8lx/sync_flag.png)
> >
> > Illustration 2: https://s32.postimg.org/vc5y9yz4l/sync_flag2.png
> >
> >
> > #Explanation of reasons:
> >
> > **Improve mining decentralization**
> >
> > One factor driving centralization is the imperative miners have to
> achieve
> > low latency in receiving and validating blocks. To achieve low latency,
> it
> > helps a lot if you have expensive low-latency internet connections,
> curated
> > network topologies, and large pools that have a plausible chance of
> finding
> > consecutive blocks. If miners are expected (or forced) to validate a
> block
> > prior to mining on top of it, the rational end game would be to outsource
> > the validation step to a trusted third party specialist who can choose
> > optimal locations on the globe to serve their (multiple?) mining pool
> > clients. These are all less decentralized than the mining situation
> Satoshi
> > and others imagined.
> >
> > **Reduce variance in mining revenue**
> >
> > Currently, there are about 144 opportunities per day for a pool or solo
> > miner to see any revenue at all. With sync flags, that number doubles to
> > 288. Sync flags are only worth 25% of what a block is worth, but this
> still
> > represents a significant reduction in variance. This variance is one
> factor
> > causing solo miners to group into pools, and large pools to be more
> > attractive than small pools.
> >
> > **Reduce or eliminate SPV mined blocks**
> >
> > One way miners have sought to make
> > full-block-transmission-and-validation-latency irrelevant has been
> through
> > "SPV" mining or "Head-first" mining. There is some evidence that these
> > techniques may be widely used, and that badgering the miners about it is
> an
> > ineffective strategy to stop them.
> >
> > In SPV mining, a miner would simply accept any block header that shows
> the
> > correct proof of work. All other validation is entrusted to other miners.
> > This practice is quite dangerous as the SPV miners can wander off on some
> > invalid chain, taking SPV nodes with them. If this occurs during a soft
> > fork, these blind miners can also fool unupgraded fully validating nodes
> > into following them.
> >
> > "Head-first" mining means that miners start hashing as soon as they
> receive
> > the block header with the correct POW, but they simultaneously validate
> the
> > block, and abandon it if is not valid. I consider this to be pretty
> safe, as
> > it strictly limits the length of an invalid chain that can result from
> > mining without validating. However, "Head-first" mining can plausibly
> > generate 2 or 3 confirmations of an invalid block. It would be nice if
> such
> > confirmations did not happen.
> >
> > The sync flag technique is similar to head-first mining, but rather than
> > hashing the next block while they wait for transmission and validation of
> > the prior block, they hash the sync flag. Nodes can differentiate between
> > sync flags and blocks, and can ignore sync flags when counting
> > confirmations.
> >
> > **Reduce or eliminate empty blocks, smoothing out resource usage**
> >
> > Empty blocks are another consequence of SPV or Headfirst mining, because
> the
> > miner cannot safely include any transactions in the block they are
> hashing
> > until they have validated the prior block. By delaying the start of
> hashing
> > the next block until after validation, miners would not have this reason
> to
> > mine empty blocks.
> >
> > **Reduce or eliminate the latency bottleneck on throughput**
> >
> > Centralization pressure due to latency issues has been a major
> preoccupation
> > over the last year. If latency mattered much less, it could represent a
> > scalability improvement that could enable higher throughput.
> >
> > **Make transaction stuffing by miners be either obvious or costly**
> >
> > Currently, the entire block reward goes to the miner who mines it. One
> > unfortunate consequence of this is that it does not cost the miner
> anything
> > to covertly stuff the block with transactions. These transactions would
> pay
> > fees and be indistinguishable from ordinary transactions, but the fees
> are
> > paid by the miner and then immediately returned to the miner.
> >
> > With sync flags, the miner must share these transaction fees with the
> > address contained in the sync flag 10 blocks prior. This means that if
> the
> > miner gives the transactions a normal looking fee, they will incur a cost
> > that will be paid to the sync flag. If the miner wants to avoid this,
> they
> > must give their stuffing transactions a zero fee, which provides evidence
> > that they are stuffing.
> >
> > Also, when miners stuff with transactions using a zero fee, they cannot
> > manipulate the perception of how much fee it takes to get into a block.
> >
> > Note that miners could still try to covertly stuff blocks that will pay a
> > sync flag that they themselves created. if this is a big concern, it can
> be
> > addressed by forcing blocks to pay multiple sync flags.
> >
> > **Gives miners something to do while they wait for attractive
> transactions
> > to appear**
> >
> > From the Montreal scaling workshop last year, we have [this
> > talk](
> https://scalingbitcoin.org/montreal2015/presentations/Day1/13-miles-carlsten-Mind-the-Gap.pdf
> )
> > which worried that as the block subsidy reduced and transactions became a
> > more important fraction of miner revenue, it would be rational for
> miners to
> > turn off their mining equipment for a "gap" phase after a block is
> found, to
> > allow time to pass as more lucrative transactions entered the mempool.
> >
> > I don't know whether this will actually happen. The presence of a
> suitable
> > backlog of transactions would help prevent this dynamic from emerging.
> But
> > if such idling behavior was the optima mining strategy, it could create a
> > serious vulnerability. Idle hands are the devil's workshop as the saying
> > goes, and idle miners represent a pool of inert hashpower that is
> available
> > to rent for all kinds of destabilizing purposes. It would be better to
> put
> > those miners to profitable work mining a sync flag while they wait.
> >
> > Also, this creates a more efficient price discovery mechanism for
> > transactions, because you allow transactions paying high fees time to
> arrive
> > to the marketplace, rather than take whatever anyone is offering because
> all
> > the "good" transactions got gobbled up in the prior block.
> >
> > **Can be easily done with a soft fork**
> >
> > Although a hard fork would be more efficient, sync flags could be easily
> > implemented using a soft fork by introducing the following rule:
> >
> > Every block must include a transaction which pays 25% of the block
> reward to
> > the address given by the 10th previous sync flag, and commits to the
> hash of
> > the 1st previous sync flag.
> >
> > _______________________________________________
> > bitcoin-dev mailing list
> > bitcoin-dev@lists•linuxfoundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
> >
>

[-- Attachment #2: Type: text/html, Size: 15287 bytes --]

 permalink raw reply	[|nested]
2016-07-26 12:47 [bitcoin-dev] Reasons to add sync flags to Bitcoin Moral Agent
  2016-07-26 13:51 ` Tom
  2016-07-26 22:03 ` Nick ODell

  
From: Moral Agent @ 2016-07-28 16:41 UTC (permalink / raw)
  To: bitcoin-dev

[-- Attachment #1: Type: text/plain, Size: 2691 bytes --]

If there is concern about the
block-with-valid-header-but-invalid-transactions-spam-attack, I have a
strategy using sync flags that may drastically reduce the problem.

Sync flags documented here:

https://github.com/moral-agent/sync_flags/blob/master/README.md)

The strategy to defeat the above attack is illustrated here:

https://s32.postimg.org/e94tqdqat/sync_flag_invalid_block.png

The key is to relax the requirement that a flag commit to a completely
valid block. The flag is valid if it commits to a valid block header, even
if the block body is invalid.

From the perspective of an individual miner, they can safely commence
mining a flag the moment they obtain (or discover) a valid block header.

As soon as the spam is discovered, miners can choose to either abandon the
flag and return to mining on the previous block, or they can continue
mining on the flag.

It's difficult for me to game out which of these strategies would be
preferable. My first thought is that the miners should have the incentive
to mine whichever option has the fewest miners, which should result in a
50/50 split.

However, the miners who continue mining the flag have a chance of ending up
in a situation where they mine the flag before anyone mines a valid block.
If this happens, it is sub-optimal for them. They can start mining for the
next valid block but if someone else broadcasts a valid block header they
will be in the same pickle that miners under the current protocol are: they
must either keep mining for a valid block, or SPV mine the newly arrived
block while they do validation. The third option, of mining a flag, is not
available to them, because the flag has already been mined for this cycle.

As a result of the above, it may be most rational for miners to (upon
learning that they are mining a flag on top of an invalid block) split
their hashpower unevenly between the flag and continuing to mine for a
valid block. The hashpower split reflects their estimates of the cost of
the above negative outcome. I think the split would be pretty close to
50/50, but deviations from 50/50 would not necessarily be bad. For example,
if they split 52/48, with more hashpower toward finding the valid block
instead of the flag, then that decreases the likelyhood that the flag will
be discovered before the next valid block, which is good for all of the
miners. So it's a nice positive feedback.

*****

This approach mostly neutralizes the harm done by the (currently very rare)
invalid block spam attack. As a kind of amazing side effect, the work done
to produce the spam is incorporated into the blockchain cumulative Proof of
Work, and the spammer is not paid for this contribution.

[-- Attachment #2: Type: text/html, Size: 3682 bytes --]

 permalink raw reply	[|nested]