How accurate are the Bitcoin timestamps?

How accurate are the Bitcoin timestamps?
2018-01-29 21:40 ` Tier Nolan
                   ` (2 more replies)
  
From: Neiman @ 2018-01-29 13:34 UTC (permalink / raw)
  To: bitcoin-dev

[-- Attachment #1: Type: text/plain, Size: 4891 bytes --]

First time posting here, please be gentle.

I'm doing a research project about blockchain timestamping. There are many
such projects, including the fantastic OpenTimestamps.

All of the projects essentially save some data in a block, and rely on the
block timestamp as a proof that this data existed at some specific time.

But how accurate are Bitcoins timestamps?

I didn't find any discussion or research regarding Bitcoin timestamp
accuracy (also not in the history of this mailing list). I share here a
simple analysis of timestamp accuracy, and a suggestion how to improve it.

Basic observations and questions:
-------------------------------------------
*1.* It seems to me that the timestamp is not the time that the block was
created. Ideally, it's the time that the miner started to try to mine the
block. However, as timestamps may also be used as a source of variety for
hashes, the exact meaning of its value is unclear.

If this is true, then there's a strange phenomena to observe in
blockchain.info and blockexplorer.com: the timestamps of blocks equals the
receiving times.

Am I wrong in my understanding, or is there a mistake in those websites?

*2.* Timestamps are not necessary to avoid double-spending. A simple
ordering of blocks is sufficient, so exchanging timestamps with enumeration
would work double-spending wise. Permissioned consensus protocols, such as
hyperledger, indeed have no timestamps (in version 1.0).

As far as I could tell, timestamps are included in Bitcoin's protocol
*only* to adjust the difficulty of PoW.

Direct control of timestamp accuracy:
-----------------------------------------------
The only element in the protocol that I found to control timestamp accuracy
is based on the network time concept.

The Bitcoin protocol defines “network time” for each node. The network time
is the median time of the other clients, but only if
    1. there are at least 5 connected, and
    2. the difference between the median time and the nodes own system time
is less than 70 minutes.

Then new blocks are accepted by the peers if their timestamps is
    1. less than the network time plus 2 hours, and
    2. greater than the median timestamp of previous 11 blocks.

The first rule supplies a 2 hour upper bound for timestamp accuracy.

However, the second rule doesn't give a tight lower bound. Actually, no
lower bound is given at all if no assumption is made about the median. If
we assume the median to be accurate enough at some timepoint, then we're
only assured that any future timestamp is no bigger than this specific
median, which is not much information.

Further analysis can be made under different assumptions. For example,
what's the accuracy if holders of 51% of the computational power create
honest timestamps? But unfortunately, I don't see any good reason to work
under such an assumptions.

The second rule cannot be strengthened to be similar to the first one
(i.e., nodes don't accept blocks less than network time minus 2 hours). The
reason is that nodes cannot differentiate if it's a new block with
dishonest timestamp, an old block with an old timestamps (with many other
blocks coming) or simply a new block that took a long time to mine.

Indirect control of timestamps accuracy:
--------------------------------------------------
If we assume that miners have no motive to increase difficulty
artificially, then the PoW adjusting algorithm yields a second mechanism of
accuracy control.

The adjustment rules are given in pow.cpp (bitcoin-core source, version
0.15.1), in the function 'CalculateNextWorkRequired', by the formula (with
some additional adjustments which I omit):

    (old_target* (time_of_last_block_in_2016_blocks_interval -
time_of_first_block_in_2016_blocks_interval) )/time_of_two_weeks

It uses a simple average of block time in the last 2016 blocks. But such
averages ignore any values besides the first and last one in the interval.
Hence, if the difficulty is constant, the following sequence is valid from
both the protocol and the miners incentives point of views:

    1, 2, 3,…., 2015, 1209600 (time of two weeks), 2017, 2018, 2019,….,
4031, 1209600*2, 4033, 4044, …

If we want to be pedantic, the best lower bound for a block timestamp is
the timestamp of the block that closes the adjustment interval in which it
resides.

Possible improvement:
-----------------------------
We may consider exchanging average with standard deviation in the
difficulty adjustment formula. It both better mirrors changes in the hash
power along the interval, and disables the option to manipulate timestamps
without affecting the difficulty.

I'm aware that this change requires a hardfork, and won't happen any time
soon. But does it make sense to add it to a potential future hard fork?

[-- Attachment #2: Type: text/html, Size: 5185 bytes --]

 permalink raw reply	[|nested]
2018-01-29 21:40 ` 
                   ` 
  
From: Neiman @ 2018-01-29 13:34 UTC (permalink / raw)
  To: bitcoin-dev

[-- Attachment #1: Type: text/plain, Size: 4891 bytes --]

First time posting here, please be gentle.

I'm doing a research project about blockchain timestamping. There are many
such projects, including the fantastic OpenTimestamps.

All of the projects essentially save some data in a block, and rely on the
block timestamp as a proof that this data existed at some specific time.

But how accurate are Bitcoins timestamps?

I didn't find any discussion or research regarding Bitcoin timestamp
accuracy (also not in the history of this mailing list). I share here a
simple analysis of timestamp accuracy, and a suggestion how to improve it.

Basic observations and questions:
-------------------------------------------
*1.* It seems to me that the timestamp is not the time that the block was
created. Ideally, it's the time that the miner started to try to mine the
block. However, as timestamps may also be used as a source of variety for
hashes, the exact meaning of its value is unclear.

If this is true, then there's a strange phenomena to observe in
blockchain.info and blockexplorer.com: the timestamps of blocks equals the
receiving times.

Am I wrong in my understanding, or is there a mistake in those websites?

*2.* Timestamps are not necessary to avoid double-spending. A simple
ordering of blocks is sufficient, so exchanging timestamps with enumeration
would work double-spending wise. Permissioned consensus protocols, such as
hyperledger, indeed have no timestamps (in version 1.0).

As far as I could tell, timestamps are included in Bitcoin's protocol
*only* to adjust the difficulty of PoW.

Direct control of timestamp accuracy:
-----------------------------------------------
The only element in the protocol that I found to control timestamp accuracy
is based on the network time concept.

The Bitcoin protocol defines “network time” for each node. The network time
is the median time of the other clients, but only if
    1. there are at least 5 connected, and
    2. the difference between the median time and the nodes own system time
is less than 70 minutes.

Then new blocks are accepted by the peers if their timestamps is
    1. less than the network time plus 2 hours, and
    2. greater than the median timestamp of previous 11 blocks.

The first rule supplies a 2 hour upper bound for timestamp accuracy.

However, the second rule doesn't give a tight lower bound. Actually, no
lower bound is given at all if no assumption is made about the median. If
we assume the median to be accurate enough at some timepoint, then we're
only assured that any future timestamp is no bigger than this specific
median, which is not much information.

Further analysis can be made under different assumptions. For example,
what's the accuracy if holders of 51% of the computational power create
honest timestamps? But unfortunately, I don't see any good reason to work
under such an assumptions.

The second rule cannot be strengthened to be similar to the first one
(i.e., nodes don't accept blocks less than network time minus 2 hours). The
reason is that nodes cannot differentiate if it's a new block with
dishonest timestamp, an old block with an old timestamps (with many other
blocks coming) or simply a new block that took a long time to mine.

Indirect control of timestamps accuracy:
--------------------------------------------------
If we assume that miners have no motive to increase difficulty
artificially, then the PoW adjusting algorithm yields a second mechanism of
accuracy control.

The adjustment rules are given in pow.cpp (bitcoin-core source, version
0.15.1), in the function 'CalculateNextWorkRequired', by the formula (with
some additional adjustments which I omit):

    (old_target* (time_of_last_block_in_2016_blocks_interval -
time_of_first_block_in_2016_blocks_interval) )/time_of_two_weeks

It uses a simple average of block time in the last 2016 blocks. But such
averages ignore any values besides the first and last one in the interval.
Hence, if the difficulty is constant, the following sequence is valid from
both the protocol and the miners incentives point of views:

    1, 2, 3,…., 2015, 1209600 (time of two weeks), 2017, 2018, 2019,….,
4031, 1209600*2, 4033, 4044, …

If we want to be pedantic, the best lower bound for a block timestamp is
the timestamp of the block that closes the adjustment interval in which it
resides.

Possible improvement:
-----------------------------
We may consider exchanging average with standard deviation in the
difficulty adjustment formula. It both better mirrors changes in the hash
power along the interval, and disables the option to manipulate timestamps
without affecting the difficulty.

I'm aware that this change requires a hardfork, and won't happen any time
soon. But does it make sense to add it to a potential future hard fork?

[-- Attachment #2: Type: text/html, Size: 5185 bytes --]

 permalink raw reply	[|nested]
2018-01-29 13:34 [bitcoin-dev] How accurate are the Bitcoin timestamps? Neiman
  2018-01-29 21:40 ` Tier Nolan

  2018-01-29 22:23 ` Bryan Bishop
  
From: George Balch @ 2018-01-29 21:53 UTC (permalink / raw)
  To: Neiman, Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 5458 bytes --]

The terms "simple ordering of blocks" and timestamp are essentially the
same thing.

On Jan 29, 2018 1:16 PM, "Neiman via bitcoin-dev" <
bitcoin-dev@lists•linuxfoundation.org> wrote:

> First time posting here, please be gentle.
>
> I'm doing a research project about blockchain timestamping. There are many
> such projects, including the fantastic OpenTimestamps.
>
> All of the projects essentially save some data in a block, and rely on the
> block timestamp as a proof that this data existed at some specific time.
>
> But how accurate are Bitcoins timestamps?
>
> I didn't find any discussion or research regarding Bitcoin timestamp
> accuracy (also not in the history of this mailing list). I share here a
> simple analysis of timestamp accuracy, and a suggestion how to improve it.
>
> Basic observations and questions:
> -------------------------------------------
> *1.* It seems to me that the timestamp is not the time that the block was
> created. Ideally, it's the time that the miner started to try to mine the
> block. However, as timestamps may also be used as a source of variety for
> hashes, the exact meaning of its value is unclear.
>
> If this is true, then there's a strange phenomena to observe in
> blockchain.info and blockexplorer.com: the timestamps of blocks equals
> the receiving times.
>
> Am I wrong in my understanding, or is there a mistake in those websites?
>
> *2.* Timestamps are not necessary to avoid double-spending. A simple
> ordering of blocks is sufficient, so exchanging timestamps with enumeration
> would work double-spending wise. Permissioned consensus protocols, such as
> hyperledger, indeed have no timestamps (in version 1.0).
>
> As far as I could tell, timestamps are included in Bitcoin's protocol
> *only* to adjust the difficulty of PoW.
>
> Direct control of timestamp accuracy:
> -----------------------------------------------
> The only element in the protocol that I found to control timestamp
> accuracy is based on the network time concept.
>
> The Bitcoin protocol defines “network time” for each node. The network
> time is the median time of the other clients, but only if
>     1. there are at least 5 connected, and
>     2. the difference between the median time and the nodes own system
> time is less than 70 minutes.
>
> Then new blocks are accepted by the peers if their timestamps is
>     1. less than the network time plus 2 hours, and
>     2. greater than the median timestamp of previous 11 blocks.
>
> The first rule supplies a 2 hour upper bound for timestamp accuracy.
>
> However, the second rule doesn't give a tight lower bound. Actually, no
> lower bound is given at all if no assumption is made about the median. If
> we assume the median to be accurate enough at some timepoint, then we're
> only assured that any future timestamp is no bigger than this specific
> median, which is not much information.
>
> Further analysis can be made under different assumptions. For example,
> what's the accuracy if holders of 51% of the computational power create
> honest timestamps? But unfortunately, I don't see any good reason to work
> under such an assumptions.
>
> The second rule cannot be strengthened to be similar to the first one
> (i.e., nodes don't accept blocks less than network time minus 2 hours). The
> reason is that nodes cannot differentiate if it's a new block with
> dishonest timestamp, an old block with an old timestamps (with many other
> blocks coming) or simply a new block that took a long time to mine.
>
> Indirect control of timestamps accuracy:
> --------------------------------------------------
> If we assume that miners have no motive to increase difficulty
> artificially, then the PoW adjusting algorithm yields a second mechanism of
> accuracy control.
>
> The adjustment rules are given in pow.cpp (bitcoin-core source, version
> 0.15.1), in the function 'CalculateNextWorkRequired', by the formula (with
> some additional adjustments which I omit):
>
>     (old_target* (time_of_last_block_in_2016_blocks_interval -
> time_of_first_block_in_2016_blocks_interval) )/time_of_two_weeks
>
> It uses a simple average of block time in the last 2016 blocks. But such
> averages ignore any values besides the first and last one in the interval.
> Hence, if the difficulty is constant, the following sequence is valid from
> both the protocol and the miners incentives point of views:
>
>     1, 2, 3,…., 2015, 1209600 (time of two weeks), 2017, 2018, 2019,….,
> 4031, 1209600*2, 4033, 4044, …
>
> If we want to be pedantic, the best lower bound for a block timestamp is
> the timestamp of the block that closes the adjustment interval in which it
> resides.
>
> Possible improvement:
> -----------------------------
> We may consider exchanging average with standard deviation in the
> difficulty adjustment formula. It both better mirrors changes in the hash
> power along the interval, and disables the option to manipulate timestamps
> without affecting the difficulty.
>
> I'm aware that this change requires a hardfork, and won't happen any time
> soon. But does it make sense to add it to a potential future hard fork?
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>

[-- Attachment #2: Type: text/html, Size: 6125 bytes --]

 permalink raw reply	[|nested]
2018-01-29 21:40 ` Tier Nolan

  2018-01-30  7:27     ` Peter Todd
  2018-01-30 10:53     ` Neiman
  2018-01-30 10:52   ` Neiman
  
From: Gregory Maxwell @ 2018-01-29 21:54 UTC (permalink / raw)
  To: Tier Nolan, Bitcoin Protocol Discussion

On Mon, Jan 29, 2018 at 9:40 PM, Tier Nolan via bitcoin-dev
<bitcoin-dev@lists•linuxfoundation.org> wrote:
> For check locktime, the median of the last 11 blocks is used as an improved
> indicator of what the actual real time is.  Again, it assumes that a
> majority of the miners are honest.

It would be more accurate to say that the median is not used for
improved accuracy but to mitigate a consensus incompatibility:

If the block's own timestamp were used for nlocktime and time based
nlocks were common on the network each miner would maximize their fee
income by setting the value as high as they could get away with.  What
concerned us wasn't so much that this would make the times less
accurate (though it would) but rather that it would create an
incentive for a runaway situation that could harm network stability
(e.g. with all miners cranking times against the 2hr window, then
creating pressure for miners to accept further and further in the
future; each responding to his own local incentives).

This incentive incompatibility could have been addressed e.g. by using
the prior block's time, but since the protocol doesn't require times
to be monotone (and for good reason!) the simple implementation of
that wouldn't have been a soft-fork.  The 11 block MTP worked out
nicely because the protocol already required new times to be larger
than that.

The timestamps in Bitcoin aren't intended to be particularly accurate.
They're used only for controlling the difficulty, and the adjustment
window is large enough that there isn't much distortion that can be
accomplished there.  It's not clear to me that much better can really
be done... if there were tighter time requirements in the protocol
miners would address them by running NTP which as an _astounding_ lack
of security in terms of how it is commonly deployed.  As far as I
know, I'm the only person whos ever mined blocks with their own
stratum 1 time source.

If times need to be accurate Bitcoin would need to use a rather
different design (e.g. each block would commit to the observation time
of the prior N blocks, and an iterative algorithm would solve for each
blocks time and each miners local offset).

IIRC open-timestamp calendar servers provide more precise
time-stamping under the assumption that the calendar server is
behaving correctly.


 permalink raw reply	[|nested]
2018-01-29 13:34 [bitcoin-dev] How accurate are the Bitcoin timestamps? Neiman
  2018-01-29 21:40 ` Tier Nolan
  2018-01-29 21:53 ` George Balch

  
From: Bryan Bishop @ 2018-01-29 22:23 UTC ( / )
  To: Neiman, Bitcoin Protocol Discussion, Bryan Bishop



On Mon, Jan 29, 2018 at 7:34 AM, Neiman via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> wrote:
>
> But how accurate are Bitcoins timestamps?
>

A perspective on block timestamp and opentimestamps can be found here:
https://lists.w3.org/Archives/Public/public-blockchain/2016Sep/0076.html

- Bryan
http://heybryan.org/
1 512 203 0507
2018-01-29 21:54   ` Gregory Maxwell

  2018-01-30 10:53     ` Neiman
  
From: Peter Todd @ 2018-01-30  7:27 UTC (permalink / raw)
  To: Gregory Maxwell, Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 978 bytes --]

On Mon, Jan 29, 2018 at 09:54:23PM +0000, Gregory Maxwell via bitcoin-dev wrote:
> If times need to be accurate Bitcoin would need to use a rather
> different design (e.g. each block would commit to the observation time
> of the prior N blocks, and an iterative algorithm would solve for each
> blocks time and each miners local offset).
> 
> IIRC open-timestamp calendar servers provide more precise
> time-stamping under the assumption that the calendar server is
> behaving correctly.

That is incorrect. The OpenTimestamps servers are specifically designed not to
be trusted, and thus do not make any cryptographically verifiable attestations
as to when timestamps were created.

In the future I expect to add a trusted timestamping scheme via disposable keys
to the OpenTimestamps protocol, but that work isn't yet complete:

https://lists.opentimestamps.org/pipermail/ots-dev/2017-May/000001.html

-- 
https://petertodd.org 'peter'[:-1]@petertodd.org

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

 permalink raw reply	[|nested]
2018-01-29 21:40 ` Tier Nolan
  2018-01-29 21:54   ` Gregory Maxwell

  
From: Neiman @ 2018-01-30 10:52 UTC (permalink / raw)
  To: Tier Nolan, Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 3470 bytes --]

On Mon, Jan 29, 2018 at 10:40 PM, Tier Nolan via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> wrote:

>
> Much of Bitcoin operates on the assumption that a majority of miners are
> honest.  If 50%+ of miners set their timestamp reasonably accurately (say
> within 10 mins), then the actual timestamp will move forward at the same
> rate as real time.
>

Thank you for replying. I agree that under the 50%+ assumption, timestamps
are reasonably accurately, but I fail to see a reason to make this
assumption.

I'm comfortable with the 50%+ assumption regarding ledger manipulation
(double-spending, deletion of transactions etc.). I'm much less comfortable
with it regarding timestamps manipulation.

Consider the following situation:
(1) miners are selfish,
(2) miners have a financial incentive to be dishonest.

(1) is a common state on how miners function nowadays. (2) is the case that
interests us when coming to do this analysis.

In the case of ledger manipulation, the 50%+ assumption is not because we
assume that miners are good-hearted (this violates (1)). It is there due to
an assumption that the financial damage to a miner would be bigger than the
gain in (2). This happens since a ledge manipulation may cause miners to
lose block rewards, and certainly will devaluate Bitcoin, an asset which
they possess.

In the case of timestamps manipulation, I don't see any financial damage
caused to miners. Timestamps manipulation (besides the 2016*n blocks) won't
harm the function of Bitcoin, and may even go undetected (it seems to me
that the main blockchain explorers don't track it). I don't see a
justification for the 50%+ assumption here.


>
> Dishonest miners could set their timestamp as low as possible, but the
> median would move foward if more than half of the timestamps move forward.
>
>
>> If we want to be pedantic, the best lower bound for a block timestamp is
>> the timestamp of the block that closes the adjustment interval in which it
>> resides.
>>
>
> If you are assuming that the miners are majority dishonest, then they can
> set the limit to anything as long as they don't move it more than 2 hours
> into the future.
>
> The miners could set their timestamps so that they increase 1 week fake
> time every 2 weeks real time and reject any blocks more than 2 hours ahead
> of their fake time.  The difficulty would settle so that one block occurs
> every 20 mins.
>
>
>>
>> Possible improvement:
>> -----------------------------
>> We may consider exchanging average with standard deviation in the
>> difficulty adjustment formula. It both better mirrors changes in the hash
>> power along the interval, and disables the option to manipulate timestamps
>> without affecting the difficulty.
>>
>> I'm aware that this change requires a hardfork, and won't happen any time
>> soon. But does it make sense to add it to a potential future hard fork?
>>
>
> For check locktime, the median of the last 11 blocks is used as an
> improved indicator of what the actual real time is.  Again, it assumes that
> a majority of the miners are honest.
>
>>
>>
>> _______________________________________________
>> bitcoin-dev mailing list
>> bitcoin-dev@lists•linuxfoundation.org
>> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>>
>>
>
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>
>

[-- Attachment #2: Type: text/html, Size: 5300 bytes --]

 permalink raw reply	[|nested]
2018-01-29 21:54   ` Gregory Maxwell
  2018-01-30  7:27     ` Peter Todd

  
From: Neiman @ 2018-01-30 10:53 UTC (permalink / raw)
  To: Gregory Maxwell, Bitcoin Protocol Discussion

[-- Attachment #1: Type: text/plain, Size: 728 bytes --]

On Mon, Jan 29, 2018 at 10:54 PM, Gregory Maxwell via bitcoin-dev <
bitcoin-dev@lists•linuxfoundation.org> wrote:

> On Mon, Jan 29, 2018 at 9:40 PM, Tier Nolan via bitcoin-dev
> <bitcoin-dev@lists•linuxfoundation.org> wrote:
>
>  if there were tighter time requirements in the protocol
> miners would address them by running NTP which as an _astounding_ lack
> of security in terms of how it is commonly deployed.
>

Could you say a few more words about this lack of security? Or share a link
if you have one. I know very little about NTPs.


> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-dev@lists•linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[-- Attachment #2: Type: text/html, Size: 1555 bytes --]

 permalink raw reply	[|nested]