Repository: ethereum/consensus-specs

Issue:
SSZ Union improvement proposal
The current [Union type](https://github.com/ethereum/eth2.0-specs/blob/dev/ssz/simple-serialize.md#composite-types) is not used in Phase0, and largely not yet implemented in client implementations (exception: lighthouse). In more dynamic contexts, it's could be very useful however. 

The direct motivation for this is the [Merge block format](https://github.com/ethereum/eth2.0-specs/pull/2257), in which we may see different transaction types that need to be encoded. To avoid overhead of lots of unused zero fields (SSZ does not truncate zeroes like RLP does), you would need different transaction type definitions. And how do you mix these different transactions in the same list, with a Union type.

One problem is that it takes 4 bytes per `Union` selector index (the switch between the type options), and we only really see a few type options in the common use case. This is a simple proposal to parametrize (at the type level, compile-time) the selector byte length. E.g. `Union[4, [uint256, Bitvector, uint8]]` would be **backwards compatible** with the already specified `Union[uint256, Bitvector, uint8]`. This would allow us to specify something like `Union[TRANSACTION_SELECTOR_LENGTH, [LegacyTransaction, BetterTransaction, SuperTransaction, AbstractedTransaction]]` (and yes, the selected index is part of the merkleization, no confusion between types in the merkle proof)

For the merge specifically, an alternative is to use opaque transactions (`List[byte, max_size], a.k.a. Bytes[max_size]` to encode an RLP transaction, instead of structured data). The pro is that the beacon-spec does not have to care about transaction types, the con is that we miss out on SSZ merkleization of transactions into the block-root, one of the very primary places that merkle-proofs would actually be very useful (you can construct a merkle proof about any transaction detail).

As a proof of concept, I implemented this new `Union` behavior in `remerkleable`, the SSZ library used in the python specification. See [here](https://github.com/protolambda/remerkleable/pull/9). (*Note for code reader: it represents the data as binary tree, and overlays it with views, it's focused on merkle proofs and data sharing more so than structural efficiency*).




Discussion:
This makes sense to me. Opaque bytes or empty fields seem inferior to using a Union type and reducing the byte footprint is simple enough.

Any reason the type def has selector length prior to union types?
`Union[selector_length, [types...]]` rather than `Union[[types...], selector_length]`
Learning from the mistakes of the Contract ABI, I would suggest to consider some forward compatibility.

I am not sure if this is the case yet, but if the first byte is used as length/kind, a potential option is to block out the highest bit as an indicator for "future/extended version". That is only to allow unions where the highest bit not-set currently for the first byte (`b[0] & 0x80 == 0`).
@djrtwo I think I prefer `Union[selector_length, [types...]]` because of the symmetry of the typing with `key, value` (`selcted_index, selected_value`) and because the `[types...]` parameter is constrained by the first parameter (can't have more than 256 types with `selector_length == 1`). And with long lists of types it doesn't end up at the very end:
```
class Block(Container):
  transactions: Union[TRANSACTION_SELECTOR_SIZE, [
    BasicTransaction,
    SpecialTransaction,
    FoobarTransaction,
    AnotherTransaction
  ]]
```
For more similarity with `List` and `Vector` we could instead define it as `N` reserved options, and derive the selector byte length to cover the reserved count.
Another change we could make is to parametrize it with a map, like: `Union[selector_size, {0: BasicTransaction, 0x42: SpecialTransaction}]` if we want to use specific selector bytes.


@axic Good point, indicating something is reserved for extension purposes can be part of the `Union` spec. It's not a bitfield though, so reserving just the `0xff` selector, or some small range, could be sufficient.


I dislike having to specify the selector size, it makes it a lot less elegant.

Given that it hasn't been used yet in any formal specification, can we simply reduce it to one byte now and assume that nobody will ever actually want a 4 byte selector in practice? I personally deem it very unlikely that anyone will run into it.

If we want to be super safe, we could simply compromise on two bytes.
@dankrad Good point, we can just do single-byte `Union`, nobody uses the current 4 byte version (or even implemented it). There's one exception I know of: Lighthouse uses it in their storage format, but that's outside of consensus. So let's simplify to a 1 byte selector, if we can confirm with Lighthouse? @paulhauner 
I'm fine with dropping the selector to one byte. I would prefer one-byte instead of a variable-length selector. Although variable-length is more powerful, I think single-byte adheres better to the first S (simple) of SSZ. A union with more than 256 variants would be pretty gross and I don't think we should add complexity to the simple case (a few variants) to cover this.

On the lighthouse end, we only use Union for the `Option` Rust type (which is either `None` or `Some(value)`). If we wanted to avoid a DB migration (at the expense of some technical debt) we could introduce a `LegacySszOption` type which serializes in the old way. That's fairly painless for us.

Looping in @michaelsproul as well :)
Cries in Golang 😢. No preference from our end as we will not be able to use this regardless
@rauljordan protobuf and generated Go code support something like it (can't do `Union[A, A, B]` though, because multiple same types): https://developers.google.com/protocol-buffers/docs/reference/go-generated#oneof
Alternatively, you can emulate it with a struct of pointer fields, and just check only a single field is set. No interfaces or weak typing that way. And manageable if there's only a limited amount of fields anyway.


I would vote for 1 byte selector + @axic suggestion to reserve the highest bit for potential backward compatible extension (to protobuf `varint` for example).
I.e. `Union` selector then would be 1 byte with the max value of `127`

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1017,7 +1017,7 @@ def run(self):
         "py_ecc==5.2.0",
         "milagro_bls_binding==1.6.3",
         "dataclasses==0.6",
-        "remerkleable==0.1.19",
+        "remerkleable==0.1.20",
         RUAMEL_YAML_VERSION,
         "lru-dict==1.1.6",
         MARKO_VERSION,


Test patch:
diff --git a/tests/core/pyspec/eth2spec/utils/ssz/ssz_typing.py b/tests/core/pyspec/eth2spec/utils/ssz/ssz_typing.py
--- a/tests/core/pyspec/eth2spec/utils/ssz/ssz_typing.py
+++ b/tests/core/pyspec/eth2spec/utils/ssz/ssz_typing.py
@@ -2,6 +2,7 @@
 # Ignore linter: This module makes importing SSZ types easy, and hides away the underlying library from the spec.
 
 from remerkleable.complex import Container, Vector, List
+from remerkleable.union import Union
 from remerkleable.basic import boolean, bit, uint, byte, uint8, uint16, uint32, uint64, uint128, uint256
 from remerkleable.bitfields import Bitvector, Bitlist
 from remerkleable.byte_arrays import ByteVector, Bytes1, Bytes4, Bytes8, Bytes32, Bytes48, Bytes96, ByteList