Title: Pectra_Retrospective
Category: Magicians

Author: timbeiko
Date: 2025-01-23T22:51:07.922Z

On ACDC#149, we agreed that before jumping into Fusaka planning, we should stop and reflect on Pectra. More broadly, we’ve recently seen many questions, strong criticism and novel improvement proposals related to AllCoreDevs (ACD).
To keep the conversation tractable, I’d suggest using this thread as a coordination point to share retrospectives and analyses of ACD, as well as proposals for how to improve things. While it’s fine to highlight problems without necessary having a solution to them, hopefully we can keep the conversation civil and productive 
I’ll use this top post to categorize high-quality responses from Client Teams, Infrastructure Providers & Developers, and Misc. Users / Community members.
In a few weeks, we’ll dedicate time on ACD to discuss what came out of this discussion.

While I don’t want to overly constrain how feedback is provided, here are suggested starting points for reflection:

Over the past year, what went better than expected in Pectra’s development and/or ACD generally? What caused this?
Over the past year, what went worse than expected in Pectra’s development and/or ACD generally? How could we have prevented this?
Looking forward, what should we start doing, stop doing and/or do differently?
What should we not compromise on, even if there is pressure to do so?
What parts of ACD are “legacy process debt”, which we should consider rethinking from first principles?

As we discuss this, we should deeply consider the second-order impact of proposals. For example, it’s often suggested that ACD should be “more efficient”. On the surface, no one would disagree. That said, it’s easy to see how there’s a tension between speed and other aspects of the process, such as security, openness and biasing towards work that has less “unknown unknowns”.
This isn’t to say that we shouldn’t try and make the process more efficient, but that we should be mindful of what we tradeoff to get those efficiency gains.

Author: abcoathup
Date: 2025-01-26T06:39:52.895Z

Community response
Summarized ACD calls async for 3.5 years (former editor of weekinethereumnews.com).
Collected summaries/writeups in Protocol Calls & happenings - Fellowship of Ethereum Magicians

Better
ACD summaries:
ACD call summaries of actions/decisions written by moderators on Eth Magicians, generally within 24 hours (thanks @timbeiko & @ralexstokes).
Scoping priority writeups
Client team writeups of scoping priorities: e.g. Reth, Lighthouse, Lodestar & Prysm.  Allows for considered async review/response.
ACD stream/recording:
Adding the chat to the stream/recording of ACDC.  The chat is where much of the discussion happens and contains a large amount of context.
Worse
Scoping:
Pectra was originally pitched as a small upgrade targeting end of 2024 (before Verkle), it became the largest upgrade ever (even after being split), targeting Q1 2025.
Planning started formally in January 2024 and the scope for the first half was finalized in December.  We ended up with too many EIPs being included over many months, with the rush to get in before Verkle put everything on hold. (the irony being that this delayed Verkle so much that Verkle might not be used for statelessness, and statelessness may not be the highest priority on the roadmap anymore).
Start
Roadmap:
Upgrades should be driven by a public multi-year roadmap.  This roadmap can and should be subject to change.  EIPs should generally only be considered for inclusion if they significantly move the roadmap forward.
Informal Veto:
Rough consensus should include an informal veto of EIP inclusion by:

Client teams considered to have greater than one third market share.
Currently assumed to be only Geth & Nethermind.  As an example, Geth were strongly against EOF, as they were a majority client at the time (with all the responsibility & pressure that this status entails) they should have had a veto.  If we don’t want teams to have this power then the community can remove it by improving client diversity.
The veto should only be informal, to ensure that rough consensus isn’t captured by any one participant.
DevOps.
If EthPandaOps believe they can’t safely test a combination of EIPs (or sheer number of EIPs) then those EIPs should be proposed for inclusion in a later upgrade.  As an example ethPandaOps call for a Pectra split

Stop
Saying yes
We want to accelerate upgrades, increasing the reach rather than widening the scope, so we need to stop saying yes to EIPs that don’t significantly move the roadmap forward.
We need to heavily use denied for inclusion, regardless of how much good work/effort people have put in, EIPs should be weighed against the roadmap, and hurt feelings aren’t part of this equation.
Do differently
Scoping:
At the start of each upgrade cycle, authors should propose for inclusion EIPs that move the roadmap significantly forward.  EIP authors/champions should add a 5 minute presentation (video & slides) to their EIP discussions topic in Eth Magicians.
This would allow client teams & the community to review them async. (without having to discuss on ACD).
Client teams (and the rest of the community) could writeup which EIPs they believe based on the roadmap should move to considered for inclusion or declined for inclusion.
A proposed scope of considered for inclusion and declined for inclusion can be created.  ACD can decide on the remaining EIPs where there isn’t rough consensus on CFI or DFI.  If needed those authors could take questions (depending on the number this should be a breakout).
The proposed scope can then be updated.  Client teams having the opportunity to estimate the effort involved, along with recommending any changes to CFI or DFI.  Informal vetos could also weigh in.
ACD
ACD should be the venue for (synchronous) decisions.  Outside of scoping, ACD should be short, ideally 30 minutes.  Any discussions should be time boxed (e.g. no more than 10 minutes).  If longer discussions are needed then these should move to a breakout or async.
The agenda should clearly indicate what decisions are to be made and the agenda shared widely.
New EIPs shouldn’t be presented at ACD, instead authors/champions should create slides & a five minute video to review async.
Any mic issues should bump a presenter to the next call. #micDAO
ACD chat logs & transcripts
Chat logs (where there is a lot of contextual discussion) & AI generated transcripts should be made available within 24 hours of the call.  (@nicocsgy is on the case).  These can be stored in GitHub - ethereum/pm: Project Management: Meeting notes and agenda items and replace the current manual transcripts and third party call writeups (which often take weeks to appear).  We should also copy the moderator summary from Eth Magicians to the call folder in ethereum/pm.
Not compromise
Rough consensus
Decisions are made by rough consensus.  Clients have a variety of owners, including an L2, a VC and the EF.  Other than the proposed informal veto, no ACD participant should have special status.   Roadmap, CFI/DFI should all be decided by rough consensus and avoid ACD being captured by any one group.
Scoping process
Fusaka upgrade should be treated as a clean slate, rather than waiting for Glamsterdam.  All EIPs should be reset to proposed for inclusion and reviewed against the roadmap.  PeerDAS and/or EOF may end up being the only EIPs to be considered for inclusion, but we should also look at FOCIL, ePBS etc and choose only the EIPs which will significantly move the roadmap forward.
Legacy process debt
EIP process
ACD should own the EIP process or at the very least have the process work for them.  It should be fast and easy to get an EIP to draft status.  This was one of the goals with splitting out ERCs from EIPs.
Just as ERC now has a funded role for ERC & standards coordinator, we (EF or another group such as the Protocol Guild) should be funding an EIP coordinator.

Author: abcoathup
Date: 2025-01-26T08:41:06.340Z

Paradigm – 25 Jan 25



Ethereum Acceleration
Paradigm is a research-driven crypto investment firm that funds companies and protocols from their earliest stages.

Author: ileuthwehfoi
Date: 2025-01-27T12:48:26.145Z

I think the Paradigm Ethereum Acceleration post is totally correct that ACD is a bottleneck and that client developers should not be the primary decision-makers. I don’t think this is desirable for anyone, not even the client developers, who have too much responsibility and have to constantly make controversial and stressful decisions they shouldn’t have to.
I think the main issue with Ethereum development is that we’ve erroneously felt that “sufficiently decentralized” means that the process has to be an entirely organic process. Devops came about when people adapted industrial engineering concepts to the environment of the server. We should also be able to adapt principles from devops to design efficient processes which are nevertheless still decentralized.


x.com



Nelson → TWC 🌎💻 (@nelsonmckey) on X
@nelsonmckey

@TimBeiko Maybe if I had to steelman it, that Ethereum is so complex and the research process is so organic that some majority of development planning only happens on testnets as speccable solutions emerge.

Therefore we’re at terminal velocity for a sufficiently decentralised approach.









We don’t even have to start from scratch. In my opinion, the development of Kubernetes provides a good starting point as an open source project which releases complicated yet robust software on a regular cadence, without any single party or person being in control.
I want to make three concrete suggestions:

Have a fixed, regular release schedule and cadence.
Formalize breakout rooms to be the primary venue for feature development, consideration, and prioritization.
Replace ACD with a dedicated steering committee led by an elected release coordinator.

There is a industrial engineering principle which keeps being independently developed which is that when setup for a particular process is slow, it will always have delays. Because setup is slow, the tendency is to make the batches large so that the impact of setup time becomes less. However, the large number of changes results in added complexity which results in delays. This leads to a vicious cycle where more changes are piled on, resulting in more delays, because if you miss your chance there are no guarantees for when your change can get in. The solution to this is actually to make the batches smaller. Small and simple changes are less likely to cause delays, and the large setup time provides an incentive to dedicate resources towards automation. It must be regular so people know that missing the fork does not mean the end of the world. Kubernetes releases a new version every 15 weeks. I think Ethereum should be able to do something similar: perhaps something like 10 weeks for development into a code freeze with 5-10 weeks of testing (decreasing as testing infra gets better and more automated, and possibly overlapping with the start of development for the subsequent fork). There should be no delays: if a EIP isn’t ready for testing or fails testing, it gets dropped, you can bring it back when it’s ready. Nor should any single client failing to be ready hold up everyone else.
In Kubernetes, feature development is handled by what they call Special Interest Groups (SIGs). In my opinion, ACD has no business in deciding something like which of two blob changes should be prioritized in a fork. Instead, there should be a Data Availability SIG, consisting of both researchers and developers and which is open to the public to view and contribute to. Instead of EIPs being opened by a single person who then has to shepherd it into existence (or more likely, get completely burnt out), the SIGs should have their own separate forums for the purpose of research, consensus building, and development. Something like specifying an EIP should be done entirely within the relevant SIG. For each fork, the SIG should decide which specified EIPs they want to prioritize for each fork and work with client developers (who should also be members of relevant SIGs) to ensure that they make it in. Alternatively, dedicated groups like SIGs may find that they can deliver features without requiring a hard fork: it’s my opinion if there was a rollup SIG the ball would have started rolling much earlier with interop than it did.
This results in decision-making and planning being taken out of ACD, arguably rendering it unnecessary. The main thing left is coordination, which can still be done via a regular group call, but which is probably better done informally, through temporary committees, or by a dedicated team headed by a release coordinator. Decentralization can be added by having the release coordinators be elected or selected by sortition from valid candidates, starting by shadowing the current coordinator and then rotating into the role for the next fork.
These changes would remove the bottleneck that is ACD, separate planning from execution and allow them to happen in parallel, distribute power from client developers, add transparency and predictability to the fork process, and provide support to EIP owners and newbies entering the space.

Author: timbeiko
Date: 2025-01-27T16:43:28.176Z

Thanks for sharing @ileuthwehfoi! I wasn’t aware that Kubernetes was structured this way. I’ll dig into it more!

Author: philknows
Date: 2025-01-27T21:09:16.796Z

There should be no delays: if a EIP isn’t ready for testing or fails testing, it gets dropped, you can bring it back when it’s ready. Nor should any single client failing to be ready hold up everyone else.

I’m curious how anyone sees this working in the context of Ethereum where hard forks require a simultaneous upgrade involving all clients of various speeds. Leaving one client behind creates chaos for that group of users, especially if they’re a widely used client.
I’m sure we can do better with coordinating consensus-only forks or execution-only forks, but this will still involve at least 5 teams to ship together and leaving one or two behind will only exacerbate consolidation when we need to have more client diversity due to inevitable client bugs.
In addition, depending on what is contained within the SFI’ed EIP and how much of the codebase it affects, it can be difficult to just drop a feature last minute because someone else was running behind. The interconnectedness and dependencies between some EIPs for inclusion will require utmost care and robust coordination for success, including the order of which EIPs are implemented. A lot of the problems/delays happen during the client implementation phase where we discover many unforeseen issues, and not in the proposal phase (see EIP-7549 - theoretically easy, practically difficult).
This could be made easier by enforcing smaller, regular forks as suggested, but leaving a client team behind on a hard fork or dropping/changing the scope last minute seems very impractical.

Author: rkapka
Date: 2025-01-28T07:23:49.887Z

I think the main point of this article - that we have to ship more things faster - doesn’t think about core developers in the least.
Developing Ethereum is like climbing an infinite mountain. Even if you think you are at the peak, you realize that it’s not the real thing yet and there is a higher place to climb to. And that’s fine in itself, after all the journey is more important than the end goal. Because once you reach a goal, there’s nothing more to do.
But imagine you are a hiker going up that mountain. Would you rather try to reach the “top” as quickly as possible without stopping or hike at a comfortable speed, taking breaks to sit down by the stream, eat a sandwich and listen to the sounds of nature?
Researchers and core developers are these hikers. Writing a client is not only about implementing EIPs. There are specification items outside of EIPs, client-specific features, maintenance, bug fixing, testing, tech debt, … It all takes time.

We think ossification is too risky for Ethereum. It prevents Ethereum from staying competitive as a platform, as applications and users move toward more centralized alternatives.

Bitcoin is as ossified as one can get, and it’s been at the top for the last 15 years. Ethereum is not going anywhere. Will it become less competitive if we ship less? Sure. Will another L1 overtake it? Maybe. Is it the most important thing in the world? No.
I know I am only criticizing and not proposing how we change things. I do believe that we can improve on how we approach things and I am sure many will have great ideas. I just don’t buy the “do more, and do it faster” slogan.

Author: asn
Date: 2025-01-28T12:08:08.058Z

Thanks for the thoughtful comments @ileuthwehfoi! I agree with you on many points! I’ve been trying to type this post for days; but you made it significantly easier. Here comes thoughts and lists!

I think this latest fork has shown that ACD calls, while a fun social experiment and a decent venue for technical discussions, might not be the ideal place for major governance decisions in Ethereum.
What kind of decisions does ACD typically handle?

Which features should be in the next fork initially?
When should the next fork happen?
Adding/removing features mid-cycle as development progresses
Deciding if a fork should be delayed or expedited

And now, here are a few reasons on why ACD is a poor place for decision-making. I’m sure there are more:

Too many cooks in the kitchen. This is a problem for a bunch of reasons:
a. Yapping can take valuable time and literally wastes 100 people’s time
b. Conversely, people with important things to say might stay silent because they fear taking up space, or because it might serve them better to stay silent and it’s easier to do so in a huge zoom call
c. A big group is inherently less agile: A small, focused team can often reach decisions more effectively and quickly.
Lack of accountability. The lines of responsibility are fuzzy.
Lack of structure. There is an agenda, but sometimes the zoom call needs to go on to the next topic proposed in the
github issue, and an important decision gets shelved.
Things will only get worse. As the number of developers, client teams, and stakeholders grows, the Zoom call chaos grows with it. Without a more structured governance approach, ACD calls will become even more unwieldy.

IMO, the natural step forward here is to introduce a smaller public “steering committee” that serves as the decision-making body of Ethereum, while keeping ACD as the technical dev call. I’m not gonna go deep into how we should compose, choose or name this steering committee, to avoid natural bikeshedding.
Instead, I will spend the rest of this post addressing the all-time-classic knee-jerk reaction “ACD is decentralized but a steering commitee is not!!11”:


There’s a misconception that a 100-person broadcast channel automatically gets you the “decentralization” sticker. Some even conflate decentralization with chaos and disorder, murmuring phrases like “yes kid, the process sucks, but that’s how decentralization looks like…”.
Decentralization for decentralization’s sake can lead to bad outcomes. Instead of an appeal to decentralization we should think what security properties we want from our governance body. When people chant ~decentralization~ they usually mean “No single entity should be able to control the future of Ethereum”. IMO this does not require ACD in its current shape to be achieved.


There is no reason to think of a 100 people zoom call as more secure compared to a smaller zoom call. Power imbalances, the tyrany of structurelessness, behind-the-scenes lobbying, and a bunch of other fun social phenomenons, can actually make the 100 people call more susceptible to manipulation than a smaller group.


If the increasingly inefficient ACD keeps on being the governance body of Ethereum, the natural course of action is that key decisions will be taken in backrooms and ACD will just serve as a token of decentralization. I believe that if we enshrine a reasonable and efficient governance body well-in-advance we can avoid this failure case.


Just think about it as broadcast vs tree-based aggregation. If the steering-committee includes (at least) all the team leads, then if you are at the leaf layer (just a client engineer or a poor researcher) you need to make sure you communicate your thoughts or concerns to your team lead and come to agreement. The job of the team lead is to then lucidly bring the thoughts on the decision-making body.

Author: ileuthwehfoi
Date: 2025-01-28T13:59:52.329Z

Yeah, so what I wrote is the ideal. Then there’s putting it into practice, which may be very different, particularly at the start.
For example, ideally the first fork would include only a few small features or even nothing at all. Instead, it would focus more on setting the system up with testing infrastructure and process automation. Then the people shadowing the first fork are prepared to lead the next one instead of trying to design the process and coordinate new features at the same time. But would the community accept that, or would they see it as delay and a wasted fork?
It’s similar with having an interconnected codebase. Generally speaking, the ideal batch size for new releases in software is one; the hippest startups release to prod after every single merge. For Ethereum, even if the client developers could achieve that, client operators would certainly revolt. Ideally the codebase should be completely modular, allowing for independent parallel development of the consensus and execution layers. Instead, we have codebases that are fairly coupled which probably have to fork together for the foreseeable future (until BEAM maybe?).
Still, when I describe dropping features or clients from a fork, I’m describing procedures which should not commonly occur. First off, EIP selection for each merge should take into account their degree of interconnectedness, either choosing ones that are as orthogonal as possible, or having the entire fork be dedicated to a single bundle of tightly coupled features (perhaps in a minor/major staggered release cadence, where minor forks are consensus only and major forks are execution + consensus, which gives the execution clients double the time between forks). Secondly, hard deadlines should make it easier to anticipate whether or not a specific feature is likely to make it in time, allowing for preparations like creating a contingency version of the fork with a particular feature pre-removed, along with scheduled code freezes to provide soft deadlines and buffer time.
Dropping clients should be even more rare, but I’m less worried about the consequences of that, because a regular release cycle means that client operators should be ready to update their infra at specific intervals in any case. Alongside multi-client, tooling to assist with switching clients is something which should be improved anyways.
Anyways, I’m not saying we have to cargo cult k8s development. But we should probably try to avoid the mindset that Ethereum development is entirely unique. Other complicated open source projects exist, and they have processes they run which Ethereum can learn from.
re: @rkapka, I don’t think moving fast is incompatible with reducing client developer burnout, particularly if speed gains are achieved from better organization. Currently in ACD you have to pay attention to every potential EIP, but if we split things up and encouraged codebase modularization, then someone could decide they are only interested in the advanced cryptography special interest group, leaving things like networking to those who are more into it. There should be room for both hardcore and casual developers in Ethereum.

Author: rkapka
Date: 2025-01-28T15:05:14.860Z

@ileuthwehfoi You’re right, I’m sure there are ways to make things more efficient. I will admit that I took this more personally than I should have, but I found the article very one-sided, with the word “faster” being used 11 times (I just checked), while there is 0 mention what are the trade-offs of trying to go faster.
What I wanted to emphasize is that we can’t forget about developers as human beings and not just elements of an abstract process.

I don’t think moving fast is incompatible with reducing client developer burnout, particularly if speed gains are achieved from better organization. Currently in ACD you have to pay attention to every potential EIP

ACD is max 90 minutes a week (often around 60 mins), so attending it or not will make little difference. And the agenda is known beforehand, so you can tune in only for the interesting part.

Author: timbeiko
Date: 2025-01-29T01:18:14.895Z

For archival’s sake, I’ll post a proposal I made on Twitter: “I think the highest leverage thing we can do is moving to a world where once fork N is shipped, we assume the scope of fork N+1 to be final, and anything up for debate moves to N+2.”
Specifically, I would propose that we consider the scope for Fusaka frozen, only including PeerDAS and EOF in the upgrade (and potentially the CFI’d EIP-7212) and start debating the scope for Glamsterdam as we begin implementing Fusaka.
This would both keep the scope for Fusaka manageable (PeerDAS and EOF are both huge changes!) and save us time arguing over the fork scope while we are trying to implement it. When Fusaka ships, we’d consider the scope for Glamsterdam frozen.

Author: shemnon
Date: 2025-01-29T01:51:41.089Z

Here’s an idea I wrote up last year when I was reflecting on how the current devnet process was flowing: it’s basically a draft of what could be released.
The biggest feature is to keep devnets and make them so they could be shipped “on demand,” applying one of the learnings from Agile methodoligies.
My note also includes a half baked idea aboult larger features working independently of slotting.  However, I like Tim’s idea of N+2 is open for additions and closes when it becomes the N+1 fork better than my large fork idea.



HackMD



Draftnet Driven Deployment - HackMD
Pectra's iterative approach really has solidified the use of devnets. But call them draftnets to make it clear "this is a draft of what we may ship." - and make it shippable.

Author: wminshew
Date: 2025-01-29T07:02:43.933Z

Strongly in favor of:
1/ smaller & more frequent forks (and generally learning from projects like k8s, though not necessarily adopting wholesale etc)
2/ locking fork N+1 when fork N ships
3/ regular retrospectives (such as this) after every fork
As an ~outsider to ACD, I don’t know if there’s any big pushback on making these three changes atm but they seem like a reasonable starting point to me. Can consider it an experiment and revisit in the next retrospective
Personally, I would love more information and decisions to happen asynchronously via text, but I also might just be missing the proper channels. I am actively working to get more informed and involved here, as our app/contract limitations are starting to bump up against the edge of what is possible across the ethereum+ landscape today.
Last, but certainly not least, I deeply appreciate all that you all have done and continue to do for the broader community. Thank you for your service

Author: jflo
Date: 2025-01-29T19:53:53.199Z

Can we collect some concrete examples of when rough consensus has failed?
90% of the discussion on what goes into a hardfork revolves around “will it fit”.  I’d like to be sure we’re not mis-classifying estimation difficulty as governance (prioritization) problem.

Author: sophia
Date: 2025-01-29T23:26:49.815Z

We should aim for hard forks every six months beginning with Fusaka. There are many advantages of this:

Easier to reach consensus on scope. We were still considering EIPs for inclusion in Pectra 75% of the way through its life cycle. Much of this is because not being included currently means waiting a whole year, during which the entire product and technical landscape will have changed. With a more frequent schedule, it would be easier to freeze the scope one fork ahead and ship on time because bumping a feature to the next release wouldn’t mean a long wait.
Less social pressure wrt what each fork means for the future of the protocol. Ideally each fork would include a “big” feature, but it’s fine if some don’t.
Easier to shift focus. We can decide a feature is so important that it’s the only thing we care about for the next fork, without permanently derailing everything else in the research pipeline. For example, FOCIL isn’t urgent right now, but that could quickly change. PeerDAS might already be urgent enough for us to place this level of priority on it.

It would be a big win to ship PeerDAS in September. I think this is the way.

Author: shemnon
Date: 2025-01-30T05:24:44.468Z

jflo:

Can we collect some concrete examples of when rough consensus has failed?


I would say we had rough consensus when EIP-1057 - ProgPow first got the green light.
Why did consensus fail? I don’t think we had a wide enough audience attention to the decision.  Maybe we should have waited a meeting cycle before declaring consensus.  But the real opposition formed away from the regular attendees of ACD.
I’m not so sure waiting an ACD with the declaration “we will finalize on adding EIP-1057 to the next fork” because things didn’t go south in earnest until a startup ASIC project de-cloaked and opposed it, I don’t know if they would have gone public unless it was committed.
After that, and eliding a whole lot of details, consensus moved slowly but only in one direction: don’t do it.
Moral of the story: rough consensus is not a contract written in stone (at least not until the fork activates).

Author: james-prysm
Date: 2025-01-31T16:38:09.242Z

Instead of EIPs being opened by a single person who then has to shepherd it into existence (or more likely, get completely burnt out), the SIGs should have their own separate forums for the purpose of research, consensus building, and development.

(This is a person perspective and perhaps not from my team.)
Even though it’s proposed that these SIGs should include both relevant researchers and client devs, I feel like this might be a dangerous direction to go, the nice part of EIP leads shepherding is it makes the case on each team that this is important and there should be dedication to that item. If a relevant client team does not have the resources or interest to participate in the SIG wouldn’t they be alienated from that agenda? we may end up with a fragmented community. perhaps that is what “modular” means, but in this case, important contributors will be left behind and eventually pushed away.

Author: ileuthwehfoi
Date: 2025-02-01T09:12:01.101Z

I’m not totally sure I understand what you are saying, but I think it’s something like: under the current system client developers can passively allow EIPs to come to them, while the SIG system would require more active engagement which would strain smaller teams.
I’ve been thinking more about what the structure of SIGs should be for Ethereum, and currently I think this is one case where copying directly from k8s would be misguided, because k8s infra is naturally modular, while Eth is both coupled and diffuse. For example, if there was a SIG State Management it would own ABI, SSZ, RLP, and different Tries, some of which are irrelevant to you as a consensus client developer. So SIGs for Ethereum should probably not be based on research categories.
Instead, I think each Special Interest Group should own and try to maximize one particular key performance indicator of Ethereum. One possible list with examples of who would own different parts of the current roadmap:

Censorship Resistance (FOCIL)
Decentralization* (SSF)
Moneyness (Issuance Curve, Marketing)
Network Stability* (State Expiry)
Privacy
Scaling (PeerDAS)
Security* (Self-Destruct, Post-Quantum)
Ship Speed (Process Design, Documentation, Devnets)
UX Applications (EOF)
UX End-users (Interop)
UX Validators (Verkle)

Notably many of these groups already exist in some form or another. EthStaker could slot directly into the UX Validators SIG. Some exist bifurcated in highbrow and lowbrow versions, like RIG and ethfinance for Moneyness. Others exist more informally, for example I understand there is a customary step where EIPs are evaluated for DoS before consideration for inclusion, which could take place more transparently in a Network Stability SIG. But some don’t exist, even though they probably should.
(*those marked with an asterisk would probably be more defensive/veto oriented rather than focused on feature creation, with a dedicated consulting location where other SIGs can ask for advice on EIPs they are considering, as well as dedicated seats in any steering committee).
First off, I think it’s entirely possible for even small teams to have members in all SIGs, as this means that there won’t be very many of them. I think basically every Ethereum developer is here because they are ideologically aligned with at least one of these goals, and therefore naturally would want to be part of a group which is working towards them.
Secondly, you can see it’s not actually necessary to have a representative in every SIG. For example, consensus client developers probably don’t need to interact at all with SIG UX Applications.
Finally, it should be noted that the purpose of SIGs isn’t to coordinate development. That is what the fork coordinator (or ACD) is for. It’s to coordinate decision-making: SIGs provide entry points for people to get involved, places to build consensus and collaborate, and unified groups to lobby for features. The intention is that the only thing a client developer has to do for the fork itself is to implement and test. As a result, it should be entirely possible for client developers to tailor their degree of involvement, including an entirely passive or purely advisory role if that is what is desired.

Author: NelsonMcKey
Date: 2025-02-03T03:11:56.846Z

Thanks for the post - just to clarify this is my attempt to present another side to the argument, not one that I fully believe. However there is some truth to it, and imo the easiest way to address this bottleneck is to have much earlier devnets, running in parallel with current process. So while we’re deploying Pectra, Testing Fusaka, we should already have a rolling devnet stood up for Glamsterdam in order to begin speccing and exploring target EIPs.
It won’t be possible to transition to this cadence immediately, but we should try!

Author: NelsonMcKey
Date: 2025-02-03T03:15:17.705Z

Two arguments being mixed together here. If an EIP isn’t ready, we should be comfortable moving it to the release after, in order to hit a regular delivery cadence. Confidence in the timing of the next release takes pressure off this decision.
However dropping clients (and whole sets of validators who run them) is a far more serious consideration and definitely not something that should become normal.

Author: marchhill
Date: 2025-02-03T16:10:56.727Z

The Nethermind team have discussed our opinions on increasing the rate at which we ship forks and thought of some ways to improve the process.
Fork Scoping
We should aim to have a provisional plan for the next 5 hard forks, mapping Vitalik’s roadmap into concrete updates. There would still be flexibility to change fork scoping, especially more distant ones, but having this provisional timeline would greatly help in deciding where to allocate resources.
For imminent forks (within a year) we should avoid adding new EIPs to the scope unless they are already implemented and tested; and the current scope is live on a devnet.
Targeting a cadence of around six months per hard fork could help to prevent forks becoming too large. Large feature forks could be alternated with smaller forks focused on UX and quality of life improvements, giving more time to test complex new features; it is also crucial to consider cross-fork dependencies and plan accordingly.
Parallelising Implementation
Having a clear plan can help to parallelise work on future forks. Once a fork is shipped, we could already have the next in devnet phase rather than starting to discuss the scope at that point.
Specs and Testing
EIP champions should have greater responsibility in ensuring their specifications are rigorous in order to avoid differences in interpretation of the spec leading to delays. They should also coordinate with the testing team to create spec tests. Formalising and testing earlier in the process would save time overall.
Adopting EIP versioning (EIP-7577) would help to coordinate implementing spec changes.
Furthermore, more resources could be allocated to testing, and client teams could be more involved.
ACD Efficiency
Client teams should do more asynchronously before ACD to make the calls more productive.
Specifically, there could be a place where client teams can review EIPs, leave opinions, and feedback. We already have Ethmagicians, but we could create a dashboard which is more structured than a forum thread. This would display which EIPs are provisionally scheduled for each fork and could be annotated by client teams with their opinions and implementation status.
Additionally, there should be greater expectations for ACD agenda items to be reviewed before the call where possible (teams could assign members to review different items), to avoid situations where no one has had a chance to review.

Author: gballet
Date: 2025-02-04T15:56:37.226Z

This is a rewording of my comments during ACD last week. The problem isn’t the fork size, it’s that forks are bloated with EIPS that seem “small enough” that including them feels like it will not cause any delay in the fork. These claims are often taken at face value.
Core developers get very busy implementing EIPs that are completely unrelated to the big picture of the roadmap, but it feels like something is being done because yet another EIPS is included.
One can add as many committees and smaller decision groups as one wants, the problem will still be the same if there is no commitment to the overall roadmap, and if frivolous EIPs keep getting scheduled.
I agree with the Nethermind team, that there should be a much higher expectation on the quality of the testing and the specification of each EIP. This is exactly what we have tried to achieve with the verkle trees/stateless effort, and yet this has played against it: people realized how complex it was and preferred (putatively) smaller EIPs that felt like we something would ship fast.
There needs to be a way to commit to long term items on the roadmap as a way to signal that work doesn’t get wasted. In other words, we need commitment from core developers and especially the research team, that it is safe to work on extremely challenging and difficult forks. Without these guarantees:

it’s impossible to plan and work in parallel - only the next fork is thought of while everything else is a “maybe”
the amount of people willing to take the risk of working on extremely ambitious forks will decrease, as the costs to do so are high and the rewards non-existent.

This is not a question of decentralization: large, centralized companies run into the same kind of issues as they grow successful.
There’s no easy way to solve these issues, but the few steps below would be a good start:

EIP champions should justify on ACD how an EIP advances the roadmap (or Increase the security of the network, obviously). That means EIP that are currently scheduled for Prague or Osaka.
Don’t change the road map because twitter panics. Any change to the roadmap should be a month-long process involving many stakeholders, probably during an interop or a major conference.
Improve communication between research and core devs. The latest example is inclusion lists (and quantum resistance a close second). While these topics do need to be researched, they have been pushed onto the core development agenda way too early, caused a lot of confusion and had a negative impact on the timelines as teams worked on building them before realizing the idea wasn’t ready for its prime.
Commit to forks long-term. This doesn’t mean that any change should be slated for a specific fork ahead of time. This means there needs to be tracks, and if the output of any of these tracks is ready (i.e. devnets running and execution specs written) then they should be shipped in the next fork. We can have 4 forks a year with this (or 0, which means teams are working on adding value to the protocol, and that’s fine).

Author: philknows
Date: 2025-02-04T16:40:29.238Z

Two arguments being mixed together here. If an EIP isn’t ready, we should be comfortable moving it to the release after, in order to hit a regular delivery cadence. Confidence in the timing of the next release takes pressure off this decision.

100%. One of the best ways to do this is to plan n+2 ahead. We can move a lot faster on fork n+1 with this in mind also to prevent a lot of rebasing/merge conflict with more fork scope certainty.

However dropping clients (and whole sets of validators who run them) is a far more serious consideration and definitely not something that should become normal.

Definitely wouldn’t be realistic, we need to do multi-layer hard forks together and not leave anyone behind ideally. However, we can also look at where we can direct some CL/EL-only upgrades where entire features aren’t blocked due to X feature that’s not ready for the n+1 hard fork.
The Lodestar team posted a retro on our blog with some of these suggestions/ideas here: Lodestar’s Pectra Retrospective and Future Fork Ideas
If you require some of that within this thread for the AI generator or persistence purposes @timbeiko , let me know and I’ll summarize a version to post in this thread.

Author: lucassaldanha
Date: 2025-02-05T09:10:04.627Z

The Teku team has shared a document reflecting on some of the points raised by Tim and also some observations from the team. The full retrospective document can be found here. Please read below a summary of our discussion:

It is impressive to see how much we have evolved in our testing capabilities, largely due to the support from EthPandaOps and the tooling they have developed. We cannot stop praising them for all their work and how much more productive we are because of their help.
It is really good to see many teams working together and collaborating towards making the protocol better. Interop events and Eth R&D workshops have been super valuable.
Scope creep has impacted our productivity and workflow and made us take longer to deliver value. We should try to have a better understanding of the features going into a fork earlier in the cycle, and converging into a stable scope sooner.
Ideally, every fork should have a clear goal (associated with the Ethereum roadmap). This can help core-devs to understand where to focus and also help us during the process of scoping and prioritizing features.
We need to rethink the way we make decisions about EIP inclusion in a fork. Ideally, we want more mature EIPs being scheduled, after teams have had time to understand their value, impact and implementation details.
The separation between ACDE and ACDC seems artificial and creates more harm than good. We believe that we should move towards having a unified call, to talk about scope, priorities and governance related to the protocol. Separate “technical” calls can be held between Execution teams and Consensus teams to discuss anything relevant just to one of the layers.

Thanks!

Author: rkrasiuk
Date: 2025-02-06T14:10:25.689Z

Reth team has gathered opinions on Pectra scoping and development internally, here is the aggregated summary our thoughts:
EIP Consideration
Currently, the EIP is considered ready when it has been summarised, the spec has been defined and sufficient number of community members have discussed it. Then the champion of the EIP brings it on a wider ACD discussion forum and it can be considered for inclusion. The crucial step that we think is missing from this process are test vectors. This would make implementation easier for every client as well as put the authors into the mindset of encoding and possibly discovering the edge cases that their change presents.
Scoping
Pectra scoping has effectively started after Dencun has landed on mainnet and lasted essentially for almost 9-10 months (until the last EIP inclusion). We are strongly opinionated that ACD collectively should:

normalize considering EIPs for inclusion for the fork after next (later referred to as fork n + 1)
freeze “major” EIPs for the fork n + 1 , by the time fork n lands on mainnet
consider committing to a fixed fork schedule (e.g. 2 forks a year)

Testing
We should define a clear set of requirements for clients to be included in devnet on launch. These must include passing all necessary engine Hive test and could include having run an interop devnet locally with 2 or more other clients.
Devnet Parallelization
At this moment, we think that devnet parallelization (simultaneously running devnets for fork n, n + 1, n + ...) is a complex change to the current ACD process and requires more discussions. While we are not against the change, we do think that it introduces a set of challenges/questions including but not limited to:

fork activation: should we run fork n + 1 devnet with changes from fork n or should it be an isolated devnet?

the former approach puts additional strain on developers to troubleshoot potential bugs from fork n on fork n + 1 devnet
the latter approach requires downstream rebase of the changes which is time consuming and might introduce additional bugs


what to do with interdependent EIPs across forks?

Summary
We think that first and foremost we should define and change the fork scoping process and are open-minded to future changes with regard to devnet testing.

Author: timbeiko
Date: 2025-02-11T21:42:01.596Z

Thank you to everyone who has shared their perspective on this thread so far!
Putting it all together, there are three overarching themes that emerge:

Increasing the frequency of network upgrades (and limiting scope to achieve this)
Agreeing on clear technical standards for EIPs at various stages of the process, from their initial introduction to AllCoreDevs to their inclusion in an upgrade and eventual deployment
Higher-order process reforms for ACD

I think we can and should make progress on all three tracks in parallel!
Starting with (1), our guiding question should be “what would it take to ship Fusaka ~6 months after Pectra”? My personal answer here is that the scope for Fusaka should be limited to the already SFI’d EIPs: PeerDAS and EOF.
While this may not be the “perfect fork”, it undoubtedly advances Ethereum’s roadmap by scaling L2 blob capacity. Additionally, PeerDAS and EOF would both meet any technical bar we’d want to impose on EIPs for consideration, as they already run on multi-client devnets.
This brings me to (2). Assuming we agree to freeze the Fusaka scope ~now, then the first step before debating the Glamsterdam’s scope should be to agree on what the technical requirements for proposed EIPs should be. Hopefully, we can agree on that relatively rapidly, leaving several months for us to discuss various proposals (and for their champions to get them up to our standards).
In parallel to this, we should have a deeper discussion about (3). I’m excited about the different proposals for more radical rethinkings of AllCoreDevs but want to make sure we don’t rush to implement a whole new system without fully thinking thought through its second order effects. Realistically, it will take many months to get people onboard with a new way of working and I think we should accept that rather than naively expect to overhaul ACD overnight.
Putting all of this together, my tentative proposal would be to:

Try and ship smaller, faster forks, starting with Fusaka
Agree on a set of minor reforms (like those mentioned above) for the Glamsterdam planning cycle in the next few weeks
Kickstart an effort around a broader ACD reform, with the expectation that more substantial process changes would be proposed for the next fork planning cycle, hopefully in ~6 months.

Author: terence
Date: 2025-02-12T15:41:50.589Z

The Prysm team has internally discussed Pectra Retro. Here is a summary of our thoughts 
Key Learnings and Observations

Underestimating complexity of EIP

EIPs can often appear simple at first glance, but their true complexity surfaces more during implementation (Example: **EIP-7549)**. This experience has taught us the importance of evaluating EIPs and client changes with greater care before making decisions on what to include. This also relates to formalizing specs earlier and having ready-testing infra would help avoid delays caused by misevaluation in spec interpretation.

EIP prioritization did not address community need

Some EIPs accepted in Pectra, in hindsight, did not address the most urgent need today—scaling. Scaling-focused EIPs like PeerDAS should have taken priority over current Pectra EIPs. Every fork may want to have a north-star theme. Reflecting on Pectra, it’s clear how crucial it is to establish a strong and focused theme for each upgrade. Clear thematic goals will help prioritize features and reduce scope creep. Each fork should have a stable scope early in the cycle.

More streamlined decision making process

Certain decisions during Pectra took longer than necessary. Relying solely on ACD calls slowed progress due to their weekly cadence. We should move towards more frequent, open discussions via breakout rooms and Discord async to keep all stakeholders informed. ACD calls should focus on final decision-making rather than being the primary discussion forum. Meanwhile, having a unified call structure for ACDE and ACDC would improve efficiency.

Testing limitations

The primary bottleneck to shipping isn’t building—it’s testing. Multiple client implementation combos and limited testing entropies make finding critical consensus bugs late in the stage. EthPandaOps is simply legendary—there’s no other way to put it—but the reality is they can’t support the testing needs of 11 + more client teams. Kurtosis has proven to be a really useful tool for devnet testing. Greater investment in testing infrastructure within client teams is crucial. Early devnets and parallel streams for layer-specific upgrades could help reduce bottlenecks, allowing testing to happen earlier and more frequently.

Sustaining client development while shipping new features

Balancing research ambitions with practical implementation limits is critical. Client teams not only develop new features but must also maintain code health. Having a known and more visible schedule - planning forks well in advance and locking down the N+1 fork scope when N ships can help reduce the burden on client teams.
Path Forward and Process Improvements
•	Incorporate More Community Feedback
Client teams and community members should have stronger avenues for sharing input, with greater transparency in how decisions are made. Anyone that wants to participate in a certain topic can be informed but there exists a deadline, it’s on you if you miss the deadline and didn’t provide your input. Creating a structured dashboard for EIP reviews and feedback could improve the process compared to scattered forum threads. Based on this feedback, as a collective we can better weigh EIP priorities.
•	Shift ACD Focus
Use ACD calls for final decisions while encouraging more efficient offline discussions and asynchronous reviews before calls. This will help avoid long presentations and in-depth technical debates during meetings.
•	Improve Testing Infrastructure
Prioritize expanding testing capacity and recruiting more qualified testers on individual client teams level to reduce bottlenecks. Define clear requirements for EIP inclusion, including passing necessary Hive tests and running interop tests with other clients.
•	Smaller and More Frequent Forks
Target a cadence of one hard fork every six months, alternating large feature forks with smaller UX and quality-of-life improvements. This schedule would reduce social pressure on each fork and provide more time for testing complex features.
•	Parallel Devnet Strategy
Start devnets for future forks earlier in the process. While testing one fork, begin preparing a rolling devnet for the next fork to explore and spec target EIPs early. This approach would minimize delays caused by late scope discussions.
•	Emphasize EIP Champion Responsibility
EIP champions should take on greater responsibility for ensuring the readiness of their specifications with sufficient spec tests, coordinating with testing teams, and justifying how their proposals advance the roadmap or improve network security.

Author: realbigsean
Date: 2025-02-13T05:04:14.198Z

Lighthouse Team Feedback

Early and Aggressive Scope Freezes
We support a stronger commitment to early scope freezes. Scoping out fork N+1 while N is being shipped, and freezing N+1 when N is shipped sounds like a good framework. Although there should be rare exceptions for critically important features.
Governance and Call Structure
We think separating “scoping/roadmap” discussions from deep technical dives could streamline the process. This would work will with the parallelization of fork N+1 scoping and fork N development. The technical calls should probably still maintain the CL/EL division, but the scope/based calls probably should not. In this model, or even in the existing model, more forceful moderation on topic direction would help.
Steering committee
We’re cautious about adding bureaucracy and believe many of the benefits could be reaped by separating scoping from technicals calls, and harsher moderation.
Frequent Forks
Although each fork introduces process overhead, we hope we can make this more efficient. Parallelizing fork N+1 scope with fork N development reduces per-fork overhead. More frequent forks will also relieve pressure on getting EIPs in for fork N if N+1 doesn’t lag as far behind.
Finding a “North Star”
Without a unifying milestone like the Merge or Withdrawals, it’s not always clear what to prioritize. We see the next big push as PeerDAS. The “beam chain” is a bit further down the road. In between there is uncertainty about which EIPs or improvements should take precedence. A bottleneck here, is the huge amount of work it is to stay abreast of all research work, and with enough depth to have informed opinions.  We believe more structured polling or active outreach to teams could help break this gridlock.
Slow-Client Delays
There is some threshold after which we must move forward without a client. This would have a major impact on users of this client and migration path for them should be made available. We will also need to weight the landscape of client diversity after this client is dropped. However when we’ve reached thresholds of “client diverse enough” with a low-enough impact on users, we should move forward.

Author: daniellehrner
Date: 2025-02-13T13:26:16.467Z

Feedback from the Besu team:

Some EIPs were not specced sufficiently and delayed the fork because several rounds of spec changes where necessary along the way
Scoping Pectra took way too long. Freezing the SFI list should happen very early in the planning process
EIP-7742 was dropped too quickly. It might make sense to revisit it in the future to allow CL only hard forks
PandaOps and testing teams are top notch
Gating devnets seems like a good idea, but wasn’t strictly enforced. Seems to still make sense to do it
Hive tests > EEST tests. Hive uncovered more bugs in our case, because EEST does not execute100% of the code path. Hive tests should be used for gating devnets.

Author: Giulio2002
Date: 2025-02-13T14:28:10.434Z

In my opinion, this hard fork was messy and full of things that do not benefit Ethereum neither in the short term or in the long term. The only improvements that are tangible is native AA and the increase of the blob count. The rest of the EIPs were unnecessary and most were only refactorings which might as well never happened.
It is clear to me there is a problem in the decision-making part of the protocol rather than in the “development”. I think the development is fine but the decision making is too fragmented.
Fragmented decision making
The biggest issue isn’t development—it’s the growing inefficiency of decision-making caused by too many people trying to make governance calls at once.


Too many visions for Ethereum

Decision-making is slow and chaotic when 100+ people are involved. Discussions often get derailed, wasting valuable time.
Large groups struggle with agility—a smaller, focused decision-making body would be faster and more effective.



As Ethereum grows the inneficiency will get worse

As Ethereum’s ecosystem expands with more developers, client teams, and stakeholders, the chaos in ACD calls will scale with it.
Without a structured governance process, decision-making will continue slowing down, leading to unnecessary delays in upgrades and forks.



A Better Path Forward
Like @asn said, Ethereum needs a *smaller, structured decision-making body handle governance, while keeping ACD focused only on technical discussions.
Without this change, ACD will become increasingly ineffective, and major decisions will be made informally outside the calls, turning ACD into a symbolic process rather than a functional one. If Ethereum is to keep evolving efficiently, governance must scale as effectively as development does.

Author: abcoathup
Date: 2025-02-14T00:15:12.031Z

Links to Pectra retrospective responses
Consensus Layer teams
Grandine




Pectra Retrospective Process Improvement


    This was Grandine’s first hard fork where we fully participated in the process, so it’s difficult for us to compare it with previous forks. However, we have a few key insights to share: 
Lack of a Clear Flagship Feature 
Pectra ended up being a hard fork with many small improvements but no strong central message, or direction. We believe hard forks should be driven by a single flagship feature, similar to how the upcoming PeerDAS fork is structured. Additional smaller changes should only be incl…
  

Lighthouse




Pectra Retrospective Process Improvement


Lighthouse Team Feedback

Early and Aggressive Scope Freezes 
We support a stronger commitment to early scope freezes. Scoping out fork N+1 while N is being shipped, and freezing N+1 when N is shipped sounds like a good framework. Although there should be rare exceptions for critically important features.
Governance and Call Structure 
We think separating “scoping/roadmap” discussions from deep technical dives could streamline the process. This would work will with the parallelization of fork N+…
  

Lodestar



ChainSafe – 4 Feb 25



Lodestar’s Pectra Retrospective and Future Fork Ideas
Key lessons learnt from Ethereum’s latest hard fork. From feature cramming to scope creep, we’re exploring ways to streamline coordination, shorten fork cycles, and enhance stability—ensuring Ethereum remains resilient and future-ready.





Nimbus
Prysm




Pectra Retrospective Process Improvement


    The Prysm team has internally discussed Pectra Retro. Here is a summary of our thoughts 
Key Learnings and Observations

Underestimating complexity of EIP

EIPs can often appear simple at first glance, but their true complexity surfaces more during implementation (Example: **EIP-7549)**. This experience has taught us the importance of evaluating EIPs and client changes with greater care before making decisions on what to include. This also relates to formalizing specs earlier and ha…
  

Teku




Pectra Retrospective Process Improvement


    The Teku team has shared a document reflecting on some of the points raised by Tim and also some observations from the team. The full retrospective document can be found here. Please read below a summary of our discussion: 

It is impressive to see how much we have evolved in our testing capabilities, largely due to the support from EthPandaOps and the tooling they have developed. We cannot stop praising them for all their work and how much more productive we are because of their help.
It is rea…
  

Execution Layer teams
Besu




Pectra Retrospective Process Improvement


    Feedback from the Besu team: 

Some EIPs were not specced sufficiently and delayed the fork because several rounds of spec changes where necessary along the way
Scoping Pectra took way too long. Freezing the SFI list should happen very early in the planning process
EIP-7742 was dropped too quickly. It might make sense to revisit it in the future to allow CL only hard forks
PandaOps and testing teams are top notch
Gating devnets seems like a good idea, but wasn’t strictly enforced. Seems to still…
  

Erigon




Pectra Retrospective Process Improvement


    In my opinion, this hard fork was messy and full of things that do not benefit Ethereum neither in the short term or in the long term. The only improvements that are tangible is native AA and the increase of the blob count. The rest of the EIPs were unnecessary and most were only refactorings which might as well never happened. 
It is clear to me there is a problem in the decision-making part of the protocol rather than in the “development”. I think the development is fine but the decision makin…
  

Geth (individual responses)




Pectra Retrospective Process Improvement


    This is a rewording of my comments during ACD last week. The problem isn’t the fork size, it’s that forks are bloated with EIPS that seem “small enough” that including them feels like it will not cause any delay in the fork. These claims are often taken at face value. 
Core developers get very busy implementing EIPs that are completely unrelated to the big picture of the roadmap, but it feels like something is being done because yet another EIPS is included. 
One can add as many committees and s…
  

Nethermind




Pectra Retrospective Process Improvement


    The Nethermind team have discussed our opinions on increasing the rate at which we ship forks and thought of some ways to improve the process. 
Fork Scoping
We should aim to have a provisional plan for the next 5 hard forks, mapping Vitalik’s roadmap into concrete updates. There would still be flexibility to change fork scoping, especially more distant ones, but having this provisional timeline would greatly help in deciding where to allocate resources. 
For imminent forks (within a year) we sho…
  

Reth




Pectra Retrospective Process Improvement


    Reth team has gathered opinions on Pectra scoping and development internally, here is the aggregated summary our thoughts: 
EIP Consideration
Currently, the EIP is considered ready when it has been summarised, the spec has been defined and sufficient number of community members have discussed it. Then the champion of the EIP brings it on a wider ACD discussion forum and it can be considered for inclusion. The crucial step that we think is missing from this process are test vectors. This would ma…
  

EthPandaOps




Pectra Retrospective Process Improvement


    There were a couple of things we tried to do with Pectra and we’d like to start the retrospective by focussing on that. We had a major shift towards local testing instead of relying on live devnets. This was achieved by the use of Kurtosis (for locally running devnets), EELS/EEST (allowing the testing team to fill the tests via EELS instead of a client implementation), Assertoor (for executing actions on local devnets). This shift to local testing allowed client teams to debug and iterate quickl…
  

Community




Pectra Retrospective Process Improvement


    Here’s an idea I wrote up last year when I was reflecting on how the current devnet process was flowing: it’s basically a draft of what could be released. 
The biggest feature is to keep devnets and make them so they could be shipped “on demand,” applying one of the learnings from Agile methodoligies. 
My note also includes a half baked idea aboult larger features working independently of slotting.  However, I like Tim’s idea of N+2 is open for additions and closes when it becomes the N+1 fork b…
  





Pectra Retrospective Process Improvement


Community response
Summarized ACD calls async for 3.5 years (former editor of weekinethereumnews.com). 
Collected summaries/writeups in Protocol Calls & happenings - Fellowship of Ethereum Magicians
Better
ACD summaries: 
ACD call summaries of actions/decisions written by moderators on Eth Magicians, generally within 24 hours (thanks @timbeiko & @ralexstokes). 
Scoping priority writeups 
Client team writeups of scoping priorities: e.g. Reth, Lighthouse, Lodestar & Prysm.  Allows for considered…
  





Pectra Retrospective Process Improvement


    I think the Paradigm Ethereum Acceleration post is totally correct that ACD is a bottleneck and that client developers should not be the primary decision-makers. I don’t think this is desirable for anyone, not even the client developers, who have too much responsibility and have to constantly make controversial and stressful decisions they shouldn’t have to. 
I think the main issue with Ethereum development is that we’ve erroneously felt that “sufficiently decentralized” means that the process h…

Author: sauliusgrigaitis
Date: 2025-02-21T08:36:19.800Z

This was Grandine’s first hard fork where we fully participated in the process, so it’s difficult for us to compare it with previous forks. However, we have a few key insights to share:
Lack of a Clear Flagship Feature
Pectra ended up being a hard fork with many small improvements but no strong central message, or direction. We believe hard forks should be driven by a single flagship feature, similar to how the upcoming PeerDAS fork is structured. Additional smaller changes should only be included in exceptional cases - ideally, only if they don’t delay the shipment of the flagship feature. This approach would help control scope and accelerate fork releases.
Stronger Requirements for Non-Flagship EIPs
Any non-flagship EIPs should only be included if the proposer has implemented them in a real client and they are significantly integrated into a few ecosystem projects. This ensures that we fully understand the impact and potential friction introduced for users. It would also help prevent seemingly simple “single-line change” EIPs that, in reality, require extensive refactoring in clients and major adjustments in dependent community projects.
Faster Fork Releases Through Parallel Development
Forks need to be shipped faster, and the only viable way to achieve this is through parallel development. PeerDAS is a great example - Pectra hasn’t even been deployed on the testnet yet, and we’ve already had multiple PeerDAS devnets. However, for this model to work, teams need to be large enough to split internally so that dedicated sub-teams can focus on individual forks without key contributors being stretched across multiple efforts.

Author: wjmelements
Date: 2025-02-27T04:39:47.365Z

sauliusgrigaitis:

Pectra ended up being a hard fork with many small improvements but no strong central message, or direction.


From the developer ecosystem, I think the flagship feature of Pectra is EIP-7702 (EOA setcode). I don’t think the other features compare. Setcode is a big improvement for personal accounts.

Author: parithosh
Date: 2025-03-05T22:39:21.202Z

There were a couple of things we tried to do with Pectra and we’d like to start the retrospective by focussing on that. We had a major shift towards local testing instead of relying on live devnets. This was achieved by the use of Kurtosis (for locally running devnets), EELS/EEST (allowing the testing team to fill the tests via EELS instead of a client implementation), Assertoor (for executing actions on local devnets). This shift to local testing allowed client teams to debug and iterate quickly, creating faster feedback cycles and for devnets to be far more stable than they used to be. Stable devnets also allow us to focus on more complicated tests that might need manual involvement. This change seems to be mostly well received and positive, so we will likely continue down this route.
In the past, we used to have a lot of bugs that were caught twice - once on Hive and again on devnets. This led to a situation where broken devnets triage cycles were diverting dev time away, especially when the bug was already caught in a more reproducible and appropriate location. To remedy this, the change was with the enforcement of test coverage before devnet inclusion. EEST tests were used on Hive as well as Assertoor to trigger the bugs locally before we spent time and effort in devnets. Client teams could also individually do so, with configs pinned in discord. Hive additionally got a major update with the UI as well as the manner of running, reducing time for test runs from ~24h down to ~4h, while also being cheaper to run. This shift has largely been positive and we will likely continue to spend time improving hive and catching bugs locally. However, this leads us to the first criticism - the enforcement of test passing means that there would likely always be a difference between the fastest and the slowest client teams. This problem is exasperated due to the increase in clients we need to test (We had 8 clients to test during the merge, now we have ~12 and the matrix of ELs and CLs to test is too large for effective local tests). One potential approach is to select and enforce all tests in the major clients, while offering the infra and configs for smaller clients (allowing the teams themselves to test their client rather than the testing team). We are currently looking for ACD input on how to proceed with the increased client load.
One of the major criticisms of the Pectra fork process has been the lack of strong commitment of what Pectra is until quite a lot of time had passed. We are in favour of earlier commitments to “core” EIPs that go into a fork and happy to expand it with tested features, allowing us more flexibility in choosing when to ship. E.g: Pectra could have been fixed with the core of maxEB, 7702/etc, with PeerDAS and EOF as strong contenders for features that could make it. Then a few months into the fork process, we could take a call on if the core EIP set is increased with PeerDAS/EOF or if those could go in the next fork. This would allow us more time thinking about blockers for the core EIP, rather than spending it on testing features pre-emptively. This point has been made by other client teams already and we won’t be expanding more on the topic, we do think the CIP/SIP + testing requirement changes to the ACD process adequately address this point. We will however start explicitly assigning a person on the pandaOps team to be the point of contact for each feature that is CFI-ed, while providing best effort support to non-CFI’ed features. ie.: As a EIP champion you can work independently on EELS/EEST/CL specs to get a feature CFI-ed by ACD, pandaOps will then assign a person for the devnet cycle/testing co-ordination and ACD can judge the feature for SFI, SFI-ed EIPs will get our highest priority support. This does indeed mean the barrier for an EIP to become CFI must be high to effectively use resources.
As outlined by the recent holesky incident, we seem to be ignoring tech debt/maintenance in lieu of new features. We would propose seeing tech debt as a feature that gets worked on along with other features - i.e, it always takes up one slot in the CFI/SFI considerations and is a part of discussions as to how many features are reasonable to CFI/SFI. For e.g, the non-finality devnet performed in Nov already helped us understand that we need to be doing more of these, the issue was dev bandwidth and the topic was benched until after Pectra went live. Ideally allocating ongoing resources would help prevent such oversights in the future.
One of the un-written process changes over the years is the devnet specific spec document. This has become a schelling point for discussions in the testing calls as well as gives us an idea of the open PRs and immediate work that needs co-ordinating across teams. This has mostly been a success for short-term issues, however the current blind spot is the longer-term work required for shipping a fork/feature. The longer-term work could include a more research-ey topic (e.g moving cell proof computation to tx sender in PeerDAS), I.e a topic that has gone through the initial research thought cycles but has not made it in as a concrete spec/PR that needs implementing. These type of tasks are currently badly captured in the spec document that the pandaOps team maintains, resulting in clunkier integration in the devnet cycle. The pandaOps team is also badly placed to understand the research co-ordination implications required to get it to the PR stage (we’re more suited for being involved in the PR → implementation/tests stage). The change we’d like to request is for a new doc/process to capture said changes and for an introduction of a technical project manager, a person who could better own the research → PR pipeline and is focussed on shipping the overall feature. This person could of course be focussing on multiple features, but the preference would be for one person per feature. This role could also be handled by the EIP champion, however their skillset might not always overlap with that of a technical project manager - we might need to handle this case by case. This should help give us a better overview on features and reduce the waiting time for implementers/testers.