There has been a lot of talk lately about making your ETH2 staking setups failure ‘proof’.
Tags: Blockchain, Failure, Solutions, Staking, Ethereum

There has been a lot of talk lately about making your ETH2 staking setups failure ‘proof’. But what are the failure cases in ETH2 to worry about? And how can I figure out which are reasonable or worthwhile to have mitigation solutions in place?

A good starter

As a good starter I recommend to read this great article recently published by Carl Beekhuize from the EF. In the following post I ‘d like to add some color to Carl’s points here and there.

The key takeaway from Carl’s blog post is that ETH2 is very forgiving in being offline, particularly if you are offline uncorrelated to other validators. So for the scenarios where you are penalized for not fulfilling your validator duties you should worry more about the more likely scenarios and where your validators are offline at the same time as others. These include besides others (in increasing severity):

Infura going down Cloud service providers failure Staking service failure Client failure

How to best mitigate correlated failures like these above?

Resilience. Practically that means: Consider running your own ETH1 endpoint and most preferably a non-geth one — as it is with 80% the majority client. Secondly, if you can’t or don’t want run your staking machine at home use a smaller and/or local hosting provider. Furthermore, be aware that staking services will run a lot of validators, so a failure of a large staking service can automatically lead to correlated failures. So if you can I encourage you to operate your own ETH2 validators. And lastly, similar to your ETH1 endpoint consider running a non-majority implementation for your beacon node and validator client as a bug in the client can affect many validators at the same time. So for the time being this most likely would mean running a non-Prysm implementation.

But what about the most likely failures like internet or power outages?

As we have seen ETH2 is very forgiving when it comes to being offline. So whether it is worthwhile to mitigate failures like internet or power outages depends on two factors:

On your local situation

The number of validators

How often and how long are the outages you have experienced in the past? Do you foresee more outages in the future? The number of validators is important to consider because your failure cost increases (at least) linearly with the number of validators you are running. Let’s do the math:

An easy way to calculate whether a mitigation solution is suitable for you is to consider the downtime over the past year. Let’s call it t_down. Now calculate the rewards for all validators rewardsAvg(2*t_down) that you should expect in avg over the next year for the time 2*t_down (take 9% apr for a rough calculation). Not only are you not getting rewards when being offline you are penalized by the same about. Therefore rewardsAvg(2*t_down) is your opportunity cost. All you need to do now is compare the opportunity cost for the specific failure case with the cost for the mitigation solution.

In case of power outage a simple mitigation solution is using a UPS. UPS stands for uninterruptible power supply and is basically a battery that is put in between the power grid and your staking machine. It acts like a buffer. If there is a power outage your staking machine will get power from the UPS and if the power doesn’t return in time it shuts down your staking machine gracefully. Besides reducing the downtime, it also redcues the risk that the validator db gets corrupted during an outage. UPS’ with high battery capacity cost about $100 and consume some additional power of around 10W. So the cost for a UPS is ~$100 + ~87kWh/year.

In case of internet connection failure a simple mitigation solution is using a dual WAN router with 4G/5G modem. You can set your regular (fiber) internet connection as the default internet connection and the device automatically switches over the cellular connection once the default one drops out. You can get your hands on such a device for about $70 and it consumes about 10W additionally. For the celluar connection you also need a data plan. Costs for the data plan depend on where you are located and the amount of data you expect to need. So the cost for a backup internet connection is ~$100 +~87kWh/year + cost for data plan.

If your opportunity cost for being offline exceeds the cost for your mitigation solution, the mitigation solution might be worthwhile for you and you should ask yourself whether it is also worth your time setting it up.

For example if your internet connection has an uptime of 99%, so about 1% down times, then the opportunity cost, given an average apr of 9% and the price of Ether $553, would be $31.85 per validator. For such an internet connection it wouldn’t make sense to have a backup connection for a single validator. But if you are running 20 validators suddenly your total opportunity cost would be $637. Then the backup internet setup becomes certainly worthwhile to consider.

Are these the worst case scenarios?

No. So far we have mostly talked about scenarios where you are penalized for not fulfilling your validator duties e.g. because your validators are offline. Now there are scenarios where your staked funds are at risk. I consider these failure cases as even worse (in increasing severity):

Getting slashed Losing access to mnemonic 3rd party getting access to mnemonic

Now, the most likely way you will get slashed is by trying to build advanced validator (failover) setups that should provide redundancy or by building custom slashing protection systems. At first this sounds counter-intuitive, but you are more likely to shoot yourself in the foot with custom failover systems or custom slashing protections as we have seen in the recent slashings on ETH2:

Staking service provider Stkr by Ankr got 10 validators slashed by running a faulty slashing protection

So don’t try to be smart (: and rather be offline for a few days than having a faulty custom solution that slashes you rather than protecting you. As we have seen ETH2 is very forgiving in being offline.

Also: Don’t run your validator keys in multiple places!

Running your validators in multiple places is a bad idea as experienced by the first validator that got slashed on ETH2

The mnemonic as known as seed phrase is the very secret that you get when generating your keys. The mnemonic is the only way to withdraw your funds later. It is also the only way to re-generate your validator signing keys if you lose them. So you really need to make sure that your mnemonic is stored at a safe place. A place that is secured against third party access (that includes especially do not store the mnemonic on a internet connected device) and environmental impact like fire, flood etc.

Also, check out my article ‘The top ten mistakes you can make’ when ETH2 staking.

As always feel free to reach out to me on Twitter @phil_eth or on the Ethstaker discord @phil.eth. Ethstaker is the warm and welcoming home for all Ethereum stakers and future stakers. If you haven’t joined so far come by and say hello ❤