Repository: bitcoin/bitcoin (language: cpp)

Issue:
doc/zmq: Note about endianness does not match reality
### Is there an existing issue for this?

- [x] I have searched the existing issues

### Current behaviour

PR #23471 added a note to [ZMQ's documentation page](https://github.com/bitcoin/bitcoin/blob/master/doc/zmq.md) saying that:

> […] 32-byte hashes are in Little Endian and not in the Big Endian format that the RPC interface and block explorers use to display transaction and block hashes.

Also:

> `| hashtx | <32-byte transaction hash in Little Endian> | <uint32 sequence number in Little Endian>`
> `| hashblock | <32-byte block hash in Little Endian> | <uint32 sequence number in Little Endian>`

However, unless I am missing something, transaction and block hashes in both ZMQ and RPC appear in the same, reversed byte order (big endian).

If this is confirmed, I would like to prepare PR for the documentation.

### Expected behaviour

Documentation would not contain references to hashes being in Little Endian.

### Steps to reproduce

Start regtest with ZMQ hashblock:

```sh
bitcoind -regtest -datadir=zmqtest -server --daemon -zmqpubhashblock=tcp://0.0.0.0:43441
bitcoin-cli -regtest -datadir=zmqtest createwallet ""
```

Run independent ZMQ client in Python:

```python
  import zmq
  import binascii

  context = zmq.Context()
  socket = context.socket(zmq.SUB)

  socket.connect("tcp://localhost:43441")
  socket.setsockopt_string(zmq.SUBSCRIBE, "hashblock")

  topic = socket.recv()
  data = socket.recv()
  seq = socket.recv()

  print(f"Topic: {topic}")
  print(f"Data:  {binascii.hexlify(data)}")
  print(f"Seq:   {binascii.hexlify(seq)}")
```

Generate block:

```sh
bitcoin-cli -regtest -datadir=zmqtest -generate 1
```

```json
{
  "address": "bcrt1qpjy0a3ply66pwpj6mp8sn36aslwr3gvdq727v6",
  "blocks": [
    "4f36a8e4e1ff6ebcc13f6bec62841e44a8b4c281f1f1346f148987735fb72e0e"
  ]
}
```

Output of ZMQ client:

```
Topic: b'hashblock'
Data:  b'4f36a8e4e1ff6ebcc13f6bec62841e44a8b4c281f1f1346f148987735fb72e0e'
Seq:   b'00000000'
```

Calculate hash of the block header to crosscheck:

```sh
  header=$(bitcoin-cli -regtest -datadir=zmqtest getblockheader 4f36a8e4e1ff6ebcc13f6bec62841e44a8b4c281f1f1346f148987735fb72e0e false)
  echo $header | xxd -r -p | sha256sum | xxd -r -p | sha256sum
```

```
0e2eb75f738789146f34f1f181c2b4a8441e8462ec6b3fc1bc6effe1e4a8364f  -
```

`0e2eb75f738789146f34f1f181c2b4a8441e8462ec6b3fc1bc6effe1e4a8364f` ← hash in natural order (output of SHA256; little endian)
`4f36a8e4e1ff6ebcc13f6bec62841e44a8b4c281f1f1346f148987735fb72e0e` ← hash in reversed order (ZMQ, RPC; big endian)

---

I assume hashes are printed by `GetHex()`. The bytes are printed in reversed order:

https://github.com/bitcoin/bitcoin/blob/55cf39e4c54da6639a8f1f7c813c2909454cada1/src/uint256.cpp#L10-L18

To ZMQ, they are also sent in reversed order:

https://github.com/bitcoin/bitcoin/blob/55cf39e4c54da6639a8f1f7c813c2909454cada1/src/zmq/zmqpublishnotifier.cpp#L220-L229

### Relevant log output

_No response_

### How did you obtain Bitcoin Core

Package manager

### What version of Bitcoin Core are you using?

28.0.0

### Operating system and version

Arch Linux, 6.12

### Machine specifications

_No response_


Discussion / hints:
For reference, @hodlinator fixed a few of these in https://github.com/bitcoin/bitcoin/pull/30526 and @ryanofsky a few more in https://github.com/bitcoin/bitcoin/pull/31596.
Agree with bug report and earlier comment. Relevant ZMQ code is [Lines 220 to 240 in zmqpublishnotifier.cpp](https://github.com/bitcoin/bitcoin/blob/c242fa5be358150d83c2446896b6f4c45c6365e9/src/zmq/zmqpublishnotifier.cpp#L220-L240) and the ZMQ documentation cited is pretty wrong or at least misleading:

> […] 32-byte hashes are in Little Endian and not in the Big Endian format that the RPC interface and block explorers use to display transaction and block hashes. [...]
>
> `| hashtx | <32-byte transaction hash in Little Endian> | <uint32 sequence number in Little Endian>`
> `| hashblock | <32-byte block hash in Little Endian> | <uint32 sequence number in Little Endian>`

The most straightforward way to think about hashtx and hashblock values is as byte arrays, not as numbers. And ZMQ and RPC both interface return the same reversed byte arrays (with bytes produced by the original hash functions in reverse order). So current ZMQ documentation is wrong to describe ZMQ and RPC interfaces as being inconsistent.

Since the RPC interface returns the byte array as strings in hex format, you can interpret these strings as hexadecimal numbers, and these are the same numbers you would get if you interpreted output of the hash functions as little-endian numbers. So current ZMQ documentation is mostly wrong describe RPC interface as using big endian format, though if you squint maybe you could say it is right because the RPC interface is sending the numbers in a big endian byte representation that it previously happened to decode from a little-endian byte representation.

The ZMQ documentation is also maybe somewhat right to describe its own output as little-endian, because similar to the RPC if you just display the ZMQ output as hex, you will see numbers that match little a endian interpretation of hash function outputs. But it is also misleading because actual bytes produced by ZMQ are definitely a big-endian not a little-endian representation of those numbers.

Best way to fix this would be to avoid using terms little endian and big endian at all and just say ZMQ outputs bytes of tx and block hashes, with bytes reversed, which is the same way they are shown in the RPC interface.

Fix patch:
diff --git a/doc/zmq.md b/doc/zmq.md
index 0a74d6eef97b8..c04f6db21b767 100644
--- a/doc/zmq.md
+++ b/doc/zmq.md
@@ -87,40 +87,69 @@ For instance:
                -zmqpubrawtx=ipc:///tmp/bitcoind.tx.raw \
                -zmqpubhashtxhwm=10000
 
-Each PUB notification has a topic and body, where the header
-corresponds to the notification type. For instance, for the
-notification `-zmqpubhashtx` the topic is `hashtx` (no null
-terminator). These options can also be provided in bitcoin.conf.
+Notification types correspond to message topics (details in next section). For instance,
+for the notification `-zmqpubhashtx` the topic is `hashtx`. These options can also be
+provided in bitcoin.conf.
 
-The topics are:
+### Message format
 
-`sequence`: the body is structured as the following based on the type of message:
+All ZMQ messages share the same structure with three parts: _topic_ string,
+message _body_, and _message sequence number_:
 
-    <32-byte hash>C :                 Blockhash connected
-    <32-byte hash>D :                 Blockhash disconnected
-    <32-byte hash>R<8-byte LE uint> : Transactionhash removed from mempool for non-block inclusion reason
-    <32-byte hash>A<8-byte LE uint> : Transactionhash added mempool
+    | topic     | body                                                 | message sequence number  |
+    |-----------+------------------------------------------------------+--------------------------|
+    | rawtx     | <serialized transaction>                             | <4-byte LE uint>         |
+    | hashtx    | <reversed 32-byte transaction hash>                  | <4-byte LE uint>         |
+    | rawblock  | <serialized block>                                   | <4-byte LE uint>         |
+    | hashblock | <reversed 32-byte block hash>                        | <4-byte LE uint>         |
+    | sequence  | <reversed 32-byte block hash>C                       | <4-byte LE uint>         |
+    | sequence  | <reversed 32-byte block hash>D                       | <4-byte LE uint>         |
+    | sequence  | <reversed 32-byte transaction hash>R<8-byte LE uint> | <4-byte LE uint>         |
+    | sequence  | <reversed 32-byte transaction hash>A<8-byte LE uint> | <4-byte LE uint>         |
 
-Where the 8-byte uints correspond to the mempool sequence number.
+where:
 
-`rawtx`: Notifies about all transactions, both when they are added to mempool or when a new block arrives. This means a transaction could be published multiple times. First, when it enters the mempool and then again in each block that includes it. The messages are ZMQ multipart messages with three parts. The first part is the topic (`rawtx`), the second part is the serialized transaction, and the last part is a sequence number (representing the message count to detect lost messages).
+ - message sequence number represents message count to detect lost messages, distinct for each topic
+ - all transaction and block hashes are in _reversed byte order_ (i. e. with bytes
+   produced by hashing function reversed), the same format as the RPC interface and block
+   explorers use to display transaction and block hashes
 
-    | rawtx | <serialized transaction> | <uint32 sequence number in Little Endian>
+#### rawtx
 
-`hashtx`: Notifies about all transactions, both when they are added to mempool or when a new block arrives. This means a transaction could be published multiple times. First, when it enters the mempool and then again in each block that includes it. The messages are ZMQ multipart messages with three parts. The first part is the topic (`hashtx`), the second part is the 32-byte transaction hash, and the last part is a sequence number (representing the message count to detect lost messages).
+Notifies about all transactions, both when they are added to mempool or when a new block
+arrives. This means a transaction could be published multiple times: first when it enters
+mempool and then again in each block that includes it. The body part of the message is the
+serialized transaction.
 
-    | hashtx | <32-byte transaction hash in Little Endian> | <uint32 sequence number in Little Endian>
+#### hashtx
 
+Notifies about all transactions, both when they are added to mempool or when a new block
+arrives. This means a transaction could be published multiple times: first when it enters
+mempool and then again in each block that includes it. The body part of the mesage is the
+32-byte transaction hash in reversed byte order.
 
-`rawblock`: Notifies when the chain tip is updated. When assumeutxo is in use, this notification will not be issued for historical blocks connected to the background validation chainstate. Messages are ZMQ multipart messages with three parts. The first part is the topic (`rawblock`), the second part is the serialized block, and the last part is a sequence number (representing the message count to detect lost messages).
+#### rawblock
 
-    | rawblock | <serialized block> | <uint32 sequence number in Little Endian>
+Notifies when the chain tip is updated. When assumeutxo is in use, this notification will
+not be issued for historical blocks connected to the background validation chainstate. The
+body part of the message is the serialized block.
 
-`hashblock`: Notifies when the chain tip is updated. When assumeutxo is in use, this notification will not be issued for historical blocks connected to the background validation chainstate. Messages are ZMQ multipart messages with three parts. The first part is the topic (`hashblock`), the second part is the 32-byte block hash, and the last part is a sequence number (representing the message count to detect lost messages).
+#### hashblock
 
-    | hashblock | <32-byte block hash in Little Endian> | <uint32 sequence number in Little Endian>
+Notifies when the chain tip is updated. When assumeutxo is in use, this notification will
+not be issued for historical blocks connected to the background validation chainstate. The
+body part of the message is the 32-byte block hash in reversed byte order.
 
-**_NOTE:_**  Note that the 32-byte hashes are in Little Endian and not in the Big Endian format that the RPC interface and block explorers use to display transaction and block hashes.
+#### sequence
+
+The 8-byte LE uints correspond to _mempool sequence number_ and the types of bodies are:
+
+   - `C` : block with this hash connected
+   - `D` : block with this hash disconnected
+   - `R` : transaction with this hash removed from mempool for non-block inclusion reason
+   - `A` : transaction with this hash added to mempool
+
+### Implementing ZMQ client
 
 ZeroMQ endpoint specifiers for TCP (and others) are documented in the
 [ZeroMQ API](http://api.zeromq.org/4-0:_start).
@@ -138,7 +167,7 @@ operating system configuration and must be configured prior to connection establ
 For example, when running on GNU/Linux, one might use the following
 to lower the keepalive setting to 10 minutes:
 
-sudo sysctl -w net.ipv4.tcp_keepalive_time=600
+    sudo sysctl -w net.ipv4.tcp_keepalive_time=600
 
 Setting the keepalive values appropriately for your operating environment may
 improve connectivity in situations where long-lived connections are silently


Test patch: