Title: EIP_7495__SSZ_StableContainer
Category: EIPs

Author: etan-status
Date: 2023-08-17T23:13:43.792Z

Discussion thread for EIP-7495: SSZ StableContainer
Background

Alternative: SSZ Union (currently unused): consensus-specs/ssz/simple-serialize.md at 67c2f9ee9eb562f7cc02b2ff90d92c56137944e1 · ethereum/consensus-specs · GitHub
Alternative: SSZ Optional (currently unused): EIP-6475: SSZ Optional
Discord: #typed-transactions (Eth R&D)

Potential use

EIP-6404: EIP-6404: SSZ transactions
EIP-6466: EIP-6466: SSZ receipts

Outstanding Issues

 2024-10-29: Tagged Profile support for serialization?, https://discord.com/channels/595666850260713488/745077610685661265/1283443027750420510

Update Log

2025-04-15: Define interaction with EIP-7916 ProgressiveList

Author: etan-status
Date: 2023-08-17T23:17:54.350Z

Showcase for how this would look for transactions: https://eth-light.xyz (desktop only, not mobile-friendly)

Author: tjayrush
Date: 2023-08-21T22:28:27.200Z

This is interesting. Thanks.
The section called “Why not a Container full of Optional[E]” ends with this sentence: “Therefore, the number of fields is constant and the Merkle tree shape is stable.” I apologize if I’m misunderstanding something, but would it be correct to add, “…the Merkle tree shape is stable, if perhaps larger.”
I understand that the shape of the tree won’t change if the number of optional fields is constant, but isn’t it true that in (half?) of the cases, the tree will be twice as big?
I’m mostly just curious, but it might help to make that explicit.

Author: etan-status
Date: 2023-08-22T10:51:10.477Z

Thanks for the comment, I have updated the PR with a more detailed description of the overhead.
While the tree will indeed be twice as big, the number of hashes required to compute the root hash of the tree only scales logarithmically. SSZ implementations typically precompute root hashes of pure zero trees, so the empty half of the tree doesn’t need to be hashed in practice. For example, if you have a tree with 32 leaves, but the last 16 leaves are empty, hash_tree_root(tree) == hash(hash_tree_root(tree.left) || hash_tree_root(tree.right)) == hash(hash_tree_root(tree.left) || zeroHashConstants[depth = 4])

Author: etan-status
Date: 2023-08-22T10:57:22.719Z

As SSZ specs live in the ethereum/consensus-specs repo, the PartialContainer spec will be introduced there once we agree on an overall approach for representing transactions / receipts (EIP-6404 / EIP-6466).
Until then, I’d like to have everything in the EIPs repo, including the PartialContainer. This is in line with prior art, for example, EIP-4895, where consensus changes are being discussed in the EIP before making it into consensus-specs.
Note that I’m still working on updating EIP-6404 / EIP-6466 to use the PartialContainer. Once we have agreement, I’ll extend remerkleable with support for PartialContainer and then create a matching consensus-specs PR that also provides tests. Likewise, will open a consensus-specs PR to change the consensus Transaction structure to match the proposed execution Transaction structure.

Author: etan-status
Date: 2023-08-28T13:38:47.548Z

https://github.com/ethereum/EIPs/pull/7495#discussion_r1307423654

Yes, a new fork could relax validity constraints, while a subsequent fork could require the field to be present again. Whether that makes sense is up to the application, but the serialization format proposed here will be stable across both directions (from optional to required and vice versa), essentially providing maximum flexibility when it comes to future design space.
In fact, an implementation could simply treat all fields as optional always, if they prefer to enforce constraints in the application layer instead of the serialization library. Another implementation may go the other direction and also push down an invariant callback to ensure that only valid combinations are accepted; e.g., a receipt can contain either an intermediate state root, or a status code, but not both at the same time. Yet another implementation may try to use union types to represent various valid combinations; point being is, none of these implementation details leak into the serialization representation.

Author: etan-status
Date: 2023-09-01T19:06:12.537Z

Renamed from PartialContainer to StableContainer to highlight the stable serialization and merkleization across spec versions.
Also, simplified the specification by getting rid of the outer T and instead directly referring to the container in those situations. The Python code also was updated to no longer require the extra @dataclass layer. Serialization, merkleization, semantics and so on did not change at all.

Author: Nashatyrev
Date: 2023-09-12T14:30:22.088Z

StableContainer look like a good idea to address the issues outlined in the Motivation section.
However it seems to me that introducing of Optional type is a matter of a separate concern.
Doesn’t the StableContainer solely solve the mentioned issues?
I.e. adding a field in a new container version would be just straightforward.
Removing the field (or rather deprecating it) would be just changing its type to something like Nothing.
Moreover with introducing the Nothing type e.g.
class Example(StableContainer[4]):
    a: uint64
    b: uint64

could be represented via a regular container:
class Example(Container):
    a: uint64
    b: uint64
    f2: Nothing
    f3: Nothing

and then removing the field a  and adding a new field c would look like
class Example(Container):
    f0: Nothing
    b: uint64
    c: uint16
    f3: Nothing

Would that approach suit the needs?

Author: etan-status
Date: 2023-09-12T14:50:47.179Z

Yeah, your example is indeed how those messages would get serialized.
Passing N to the StableContainer[N] essentially fills it up with Nothing up through N total fields. Setting N in the type metadata reduces verbosity.
Using Optional[T] instead of Nothing ensures that messages of old style (for example, in your case, messages containing a) can still be parsed. This also allows for use cases that have conditional fields, for example, as highlighted in EIP-6493, where a check_transaction_supported function defines valid combinations.

Author: Nashatyrev
Date: 2023-09-13T14:38:40.131Z

etan-status:

as highlighted in EIP-6493


Yes, this is exactly the use case I’m a bit concerned about. It’s kind of leaking of SSZ static typization
Effectively instead of declaring dedicated classes for every transaction type there is a single container with a number of Optional fields which should be cross validated.
Does it make sense to declare separate SSZ type for every Tx type, like TransactionPayloadLegacy, TransactionPayloadEip2930, etc ?
I see that it would be more verbose and probably would require more boilerplate code to handle. But it would be more type safe and less error-prone imo.



 etan-status:

messages of old style (for example, in your case, messages containing a) can still be parsed


That looks to me as a separate concern. The most obvious way seems to wrap those types with a Union (or StableUnion if one needs stable merkleization and tree paths)

Author: etan-status
Date: 2023-09-13T15:59:28.833Z

Does it make sense to declare separate SSZ type for every Tx type

Overall, even if you have separate SSZ types, you’d still need additional validation to check invariants. For example, to check that the from address is correct. Or, to check that the blob_versioned_hashes match the blob in the wrapper. Or that the to field is present for the blob transaction (it’s optional without using blob). Or that the max_fee_per_gas is >= max_priority_fee_per_gas. Note you may also need matching TransactionSignatureXyz, and somehow check that it is compatible with the payload union.
Checking valid field combinations in EIP-6493 is ~20 lines, mostly to ensure that txns retain the limitations from RLP. It could be exhaustively tested. The full list of supported combos is:

TransactionPayloadReplayable → original format, not locked to chain ID (no SSZ equivalent)
TransactionPayloadLegacyRlp → optional to, no access list, no prio fee, no blob
TransactionPayloadEip2930Rlp → optional to, yes access list, no prio fee, no blob
TransactionPayloadEip1559Rlp → optional to, yes access list, yes prio fee, no blob
TransactionPayloadEip4844Rlp → required to, yes access list, yes prio fee, yes blob
TransactionPayloadLegacySsz → optional to, no access list, no prio fee, no blob
TransactionPayloadEip2930Ssz → optional to, yes access list, no prio fee, no blob
TransactionPayloadEip1559Ssz → optional to, yes access list, yes prio fee, no blob
TransactionPayloadEip4844Ssz → required to, yes access list, yes prio fee, yes blob

There’s the problem of combinatorial explosion. For example, if you want a transaction that has a blob but no priority fee nor access list, and then another one wants no blob but wants a priority fee and no access list, and so on; that’s 8 additional different “tx types” for features that don’t have anything to do with each other. With future features such as multidimensional fees, CREATE2 transaction, different sig_hash mechanisms, and so on, one may want to move towards allowing the signer to pick the combo they want instead of being forced to select a type that supports a superset of what’s needed and then having to trick around with empty lists and default values for all the features they don’t want, like currently done in RLP.
Furthermore, you’d need some mechanism to transfer type information. For example, using an enum prefix similar to Union. However, that leads to a requirement for the verifier to know about all the enum cases and their meaning. Because new types may be introduced in the future, verifiers can’t become immutable. That’s the case even if they solely care about certain fields of the container; for example, only from, to, and value, and ignore all other fields. On the other hand, with StableContainer, that could be achieved with a followup proposal like a SparseView that includes just the bitvectors, the requested 3 fields, plus a merkle proof. The merkle proof shape is statically determinable solely by the bitvectors and the requested fields regardless of tx type, which is not necessarily given by a Union approach.
About StableUnion, would be interested to understand what you mean there and how the differences to the StableContainer are.
I’d also like to better understand more type safe and less error-prone arguments. In practice, implementations likely go for a single implementation that handles all transactions. Then, for each feature, check if it is used and, if it is, process it. The difference would be that with the TransactionPayloadXyz jungle you’d need a Generics based implementation that generates another copy of the code for each individual type (feature combination), while with the StableContainer approach you’d have a single function with runtime checks for all the features. Code size is smaller with the StableContainer, while the Generics based implementation can exclude certain invalid field combinations (the 20-line check in EIP-6493) in the serialization library rather than its usage. Code size may also have implications on ZK logic based verifiers.

Author: Nashatyrev
Date: 2023-09-15T12:08:27.837Z

etan-status:

About StableUnion, would be interested to understand what you mean there and how the differences to the StableContainer are.


Sorry - my mistake here. The present Union implementation is inherently ‘stable’ actually. For some reason I thought it has the structure similar to Container



 etan-status:

Overall, even if you have separate SSZ types, you’d still need additional validation to check invariants.


Of course there is always a lot of semantics which couldn’t be expressed with types. But whenever it’s feasible it always better to express semantics/constraints via static types in the presence of such system (SSZ in our case).
Of course I don’t treat it as an immutable rule. If you are saying there is a ‘combinatorial explosion’ then of course it probably doesn’t make sense to ‘die hard’ sticking to strong typing. However some things could be expressed in a more canonical way on my mind:
Obvious example:
    # EIP-4844
    max_fee_per_blob_gas: Optional[uint256]
    blob_versioned_hashes: Optional[List[VersionedHash, MAX_BLOB_COMMITMENTS_PER_BLOCK]]

Both fields are either present or absent. Ideally it would look like:
    # EIP-4844
    eip4844Data: Optional[Eip4844Data]

class Eip4844Data(StableContainer[N]):    
    max_fee_per_blob_gas: uint256
    blob_versioned_hashes: List[VersionedHash, MAX_BLOB_COMMITMENTS_PER_BLOCK]




 etan-status:

I’d also like to better understand more type safe and less error-prone arguments. In practice, implementations likely go for a single implementation that handles all transactions.


I’m not sure about transaction representation. Teku has dedicated Java types for every hardfork version of every structure. That seriously helps to avoid shooting your foot when adding/changing processing logic across the whole codebase.
Worth to mention there is a type hierarchy as well (which is not applicable in spec) which makes things a lot simpler.

Author: sopia19910
Date: 2023-09-30T17:56:22.830Z

Does Optional[T] indicate whether the chunk has data or not?

Author: etan-status
Date: 2023-09-30T19:19:13.463Z

The present Union implementation is inherently ‘stable’ actually.

No, the existing Union is not stable. As you mention Teku, let’s do a consensus example. If you want to build an application that tracks whether a particular validator is slashed in a trust-minimized way using merkle proof of the response, you will have to keep updating your application several times a year to teach it about all the various hard forks, including all those that add completely unrelated features. This is, because the Merkleization of BeaconState is not stable across consensus-fork versions – whenever the total number of fields reaches a new power of two, generalized indices change, and therefore, the Merkle proof changes. Same when a field is removed from one version to the next. With StableContainer, the proof remains forward-compatible even if fields get added or deprecated over time.

Both fields are either present or absent.

Yes, in the business logic. But that doesn’t need to leak into the serialization or the merkleization. Adding more wrapping than strictly required in SSZ leads to overhead when using variable-length fields, with additional offset tables getting serialized.

Teku has dedicated Java types for every hardfork version of every structure.

Still possible. One can build dedicated types for all the possible subsets on top of StableContainer.
For example, back to the transactions, one could go with two types:
class BasicTransactionPayload(StableSubset(TransactionPayload)):
    nonce: uint64
    max_fee_per_gas: uint256
    gas: uint64
    to: Optional[ExecutionAddress]
    value: uint256
    input_: ByteList[MAX_CALLDATA_SIZE]
    access_list: Optional[List[AccessTuple, MAX_ACCESS_LIST_SIZE]]
    max_priority_fee_per_gas: Optional[uint256]

class BlobTransactionPayload(StableSubset(TransactionPayload)):
    nonce: uint64
    max_fee_per_gas: uint256
    gas: uint64
    to: ExecutionAddress  # Required in blob transaction
    value: uint256
    input_: ByteList[MAX_CALLDATA_SIZE]
    access_list: Optional[List[AccessTuple, MAX_ACCESS_LIST_SIZE]]
    max_priority_fee_per_gas: Optional[uint256]
    max_fee_per_blob_gas: uint256
    blob_versioned_hashes: List[VersionedHash, MAX_BLOB_COMMITMENTS_PER_BLOCK]

Both would serialize and merkleize as the TransactionPayload StableContainer. But inside business logic you can map them to concrete types for the type safety, reducing runtime checks and so on. Note that the memory layout could also be shared, with both BasicTransactionPayload and BlobTransactionPayload being a subset of TransactionPayload. One could implement that with zero copying by having an inner private payload and then define accessors into it. That also helps in consensus, because fork transitions are no longer expensive: simply set all the deprecated fields to None, then cast to the new fork and initialize the new fields.
I think it would be useful to have something like StableSubset specced out and in Python as well — What EIP-7495 describes is solely a forward-compatible serialization and merkleization scheme, it shouldn’t restrict how one prefers to write code.
Note that ProtoBufs follows a similar approach in the encoding of OneOf. The usage appears like a union, but serialization is same as a series of optional, with the parser enforcing that only one of the options is set. This means that a client that is only interested in certain aspects can still parse messages from newer servers, without having to continuously update (forward-compatibility). It also means that a server can still parse messages from old clients, without having to convert them (backward-compatibility).

Author: etan-status
Date: 2023-09-30T19:20:45.255Z

A BitVector[N] is pre-pended to the data that indicates for each field whether it is present or not. If there is a 1, the corresponding field has data. If there is a 0, the corresponding field is None.
The hash_tree_root of the BitVector[N] is also mixed into the root of the object, providing the same information for systems relying on Merkle proofs.

Author: sopia19910
Date: 2023-10-01T02:15:05.431Z

Is it correct that SSZ StableContainer refers to a Merkle tree associated with the reserve pool in the Ethereum 2.0 chain?

Author: etan-status
Date: 2023-10-29T18:02:27.566Z

I have extended EIP-7495 with a section describing an additional type safety layer built on top of StableContainer: PR
# Serialization and merkleization format
class Shape(StableContainer[4]):
    side: Optional[uint16]
    color: uint8
    radius: Optional[uint16]

# Valid variants
class Square(Variant[Shape]):
    side: uint16
    color: uint8

class Circle(Variant[Shape]):
    radius: uint16
    color: uint8

class AnyShape(OneOf[Shape]):
    @classmethod
    def select_variant(cls, value: Shape, circle_allowed = True) -> Type[Shape]:
        if value.radius is not None:
            assert circle_allowed
            return Circle
        if value.side is not None:
            return Square
        assert False

In code, could pass around Square / Circle for type-safety. While both of them would serialize / merkleize as Shape. Example usage

Why should we care about stable serialization / merkleization?

Someone may create a client today that is solely interested in the color of the shapes. They use a hypothetical API that provides them with just the color + Merkle proof that it is actually the shape’s color.
This client should not break, just because a future software upgrade adds rectangles with a width/height instead of side.

In context of Ethereum: A decentralized staking pool should not have to continuously issue software updates just because the location where information about slashing status of a validator and their balance is stored may change between forks. It should be possible to build forward-compatible clients that only require an update if actually used functionality changes. Likewise, a smart contract validating that someone paid their bill using a SSZ transaction / receipt merkle proof shouldn’t have to upgrade their contract just because a future transaction type adds an access list or a priority fee field – both those features are irrelevant for the smart contract’s purpose.

Author: etan-status
Date: 2023-10-29T18:06:54.886Z

Example usage in EIP-6493:

Update EIP-6493: Use `Variant[S]` for type safety by etan-status · Pull Request #7939 · ethereum/EIPs · GitHub

Author: etan-status
Date: 2024-04-15T16:52:41.448Z

Added compact serialization for Variant, so that this is also usable for consensus:

Update EIP-7495: Compact serialization for `Variant[S]` by etan-status · Pull Request #8436 · ethereum/EIPs · GitHub

Author: etan-status
Date: 2024-04-16T12:14:55.973Z

Added EIP for transitioning consensus SSZ data structures to StableContainer as well:

EIP-7688: Forward compatible consensus data structures

Author: etan-status
Date: 2024-04-17T13:16:10.247Z

HackMD



Next steps in the Purge - HackMD
# Next steps in the Purge  One of the less well-known EIPs in the recent Dencun hard fork is [EIP-67

Author: wemeetagain
Date: 2024-04-30T18:44:43.212Z

Overall this is a nice addition to SSZ and accomplishes its objectives.
As far as formatting of the EIP, it looks like care was given to ensure that large sections can be copy/pasted directly into the ssz document in the consensus specs. Good job there.
I do wish that the Variant type was mentioned earlier in the document, given its importance. And that it had a format that was similarly able to be copied to the ssz document.
Also feels like some sort of motivation for Variant is missing. eg:

While StableContainer provides a flexible structure for forward and backward compatibility of ssz containers, in practice, having known versions of containers, with known subsets of fields are useful, eg: a datastructure where a field is available after a fork boundary but not before. To provide this type safety as well as more efficient serialization, Variant[S] is defined.

Author: etan-status
Date: 2024-05-02T05:25:40.128Z

Have added a Variant mention in the abstract as well: Update EIP-7495: Add `Variant[S]` to abstract by etan-status · Pull Request #8511 · ethereum/EIPs · GitHub
The Variant used to share serialization as well, because for EIP-6493 it is never serialized. Only EIP-7688 prompted the more compact serialization. Agree that with its increased relevance, it deserves an earlier mention.

Author: etan-status
Date: 2024-05-04T23:44:30.876Z

remerkleable implementation (Python)
nim-ssz-serialization implementation (Nim)

Author: etan-status
Date: 2024-05-14T20:52:01.615Z

Updated to rename Variant to MerkleizeAs to better describe what it does. Furthermore, improved explanations and rationale for more clarity, and extended on tests.

Update EIP-7495: Rename `Variant` -> `MerkleizeAs` and improve rationale by etan-status · Pull Request #8558 · ethereum/EIPs · GitHub

Thanks @wemeetagain for the crazy amount of help. Feel free to also link the TypeScript implementation here

Author: wemeetagain
Date: 2024-05-15T07:05:11.373Z

Typescript implementation here:
https://github.com/ChainSafe/ssz/pull/373

Author: wemeetagain
Date: 2024-05-15T07:39:14.415Z

Looks like Update EIP-7495: Rename `Variant` -> `MerkleizeAs` and improve rationale by etan-status · Pull Request #8558 · ethereum/EIPs · GitHub got closed after I commented, pasting here for discussion:
IMO we should consider removing MerkleizeAs[“a container”] if the only benefit here is to support field reordering.
Also I’m just if reordering of fields is helpful in general, or if its just asking too much from ssz. Is there a case where reordering of fields during serialization is required, and not just a nice-to-have?
Removing the bitvector for serialization by creating a “subtype” is helpful and has tangible benefit. It follows logically from motivation to StableContainer definition. It makes MerkleizeAs serialization more efficient and on-par with Containers.
But reordering of fields feels more handwavey and wrt spec, looks more like a feature ‘because we can’, something that ‘falls from the implementation’. My recommendation would be keeping this EIP more restricted (removing all field reordering). And adding an additional EIP that extends functionality of MerkleizeAs, or defines a new thing with that functionality – where the functionality is matched by the motivation.

Author: matthewkeil
Date: 2024-05-15T12:26:49.142Z

I love the idea for StableContainer!!!  I have to admit thought naming convention for MerkleizeAs feels… less than ideal.  How do you feel about StableContainer for the abstract over-arching container and InvariantContainer for the specific, unchanging, version of instead of MerkleizeAs.  Code could look like InvariantContainer<BeaconState, Electra> or similar.  What do you think @etan-status ?

Author: etan-status
Date: 2024-05-16T13:23:28.124Z

Thanks for the feedback; adjusted the name to Profile[B].
Furthermore, the spec was simplified:

Reordering of fields in Profile[B] no longer allowed
StableContainer[N] now requires everything to be Optional[T] to avoid duplicating logic from Profile[B]
Profile[B] now require B to be a StableContainer[N], for the cases that used Container the semantics are the same when not linking the types.

https://github.com/ethereum/EIPs/pull/8562/files

Author: wemeetagain
Date: 2024-05-23T13:02:43.943Z

ssz_generic test vectors for this EIP added here:
https://github.com/ethereum/consensus-specs/pull/3777

Author: etan-status
Date: 2024-06-04T18:56:03.618Z

Added checks to remerkleable that ensure compatibility that only valid types can be created (no re-ordering, compatible field types, no extra fields, and so on).
https://github.com/ethereum/EIPs/pull/8619/files
Tests can be found here:

EIPs/assets/eip-7495/tests.py at master · ethereum/EIPs · GitHub
EIP 7495: Add ssz generic tests for StableContainer and Profile by wemeetagain · Pull Request #3777 · ethereum/consensus-specs · GitHub
https://github.com/ethereum/EIPs/blob/master/assets/eip-6493/convert_tests.py

Author: kasey
Date: 2024-06-12T23:26:40.176Z

The serialization spec defines:
def is_active_field(element):
    return not is_optional(element) or element is not None

In the specification of StableContainer[N] it’s stated that:
" All fields of a StableContainer[N] MUST be of of type Optional[T]. Such fields can either represent a present value of SSZ type T, or indicate absence of a value (indicated by None). The default valueof an Optional[T] is None."
This makes the first part of is_active_field seem ambiguous - what does is_optional mean? If it means “of type Optional[T]”, this is true for all fields by definition, so do we actually want this?
def is_active_field(element):
    return element is not None

Author: etan-status
Date: 2024-06-13T06:05:57.793Z

That’s indeed correct! Have integrated the simplification:

Update EIP-7495: Remove redundant `is_active_field` helper by etan-status · Pull Request #8665 · ethereum/EIPs · GitHub

Author: etan-status
Date: 2025-04-15T04:00:59.301Z

EIP-7916 defines a way how to get rid of capacities for List to avoid excessive hashing when the capacity is set to some arbitrary large value exceeding theoretical maximum (essentially ‘unlimited’ / application defined limit). I have updated StableContainer specs to support ProgressiveList as well.

Update EIP-7495: Define interaction with EIP-7916 `ProgressiveList` by etan-status · Pull Request #9643 · ethereum/EIPs · GitHub

We should consider whether a similar approach can also be used to get rid of the StableContainer[N] capacity – it is very difficult to reason about the N value, and chances are that devs using StableContainer may just pick a value that’s either too optimized (and later breaks when it is extended), or that’s too large (and, therefore, inefficient to hash).
The open question on adopting progressive approach for StableContainer would be in serialization / merkleization of the BitVector[N] mix-in (if N is unknown). A simple solution could be to just limit the number of fields to 254 and add another 1-byte prefix indicating N of the highest used bit plus 1. That would keep the merkleization mix-in simple (a single 256-bit node).
Alternatively we could define a ProgressiveBitlist and use that (at higher hashing cost, though!). Or we add a Bitvector to each recursive tree. First stablecontainer field gets paired with a Bitvector[1], the next 4 fields get paired with a Bitvector[4], the next 16 get paired with a Bitvector[16] and so on. Some middleground that would not introduce a 254 field limit.