Polkadot vs. Cosmos vs. Ethereum 2.0 — for real idiots
By: Patrick Wieth
Tags: Polkadot, Cryptocurrency, Blockchain, Cosmos Network, Ethereum

Main Part — Understanding the basics

We managed our way through this first passage and now come to the part how these things are generally addressed. But what does generally mean? The thing is, that no project just does everything on its own. All of them stand on the shoulders of giants and are influenced by each other. Ethereum was such a big success, that there is no project that does not look at it.

1. Scalability

There are 2 types of scaling. Vertical and horizontal scaling. This does not apply to blockchain only, it also applies to software development in general and it also applies to everything. May it be businesses or whatever processes you imagine. Let’s take out our excavators again. If we scale horizontally, then we put more and more excavators on the field. If we scale vertically, we buy faster, bigger, better excavators. Their arms move faster and have bigger shovels, nice. For such excavators it’s quite obvious that horizontal scaling implies more friction, they are more likely to interfere with each other so that the scaling is no longer linear. Vertical scaling does not have this problem in the same way, since it is just the same unit being upgraded, so no additional communication costs and no space limitation (as long as the excavator fits on the construction site :D). The problem with vertical scaling lies in the circumstance that there are technological limits. At some point it becomes complicated to find a bigger hydraulic system, that is still able to move the shovel. Once you have put the biggest hydraulic system from the market into your excavator, you cannot further scale it vertically. You can buy more hydraulic systems and operate these in parallel, this is horizontal scaling and the excavator might still be a single unit and being scaled vertically, but the hydraulic system will be subject of diminishing returns, or say sub-linear scaling (typical for horizontal scaling). So internally the excavator is scaled up horizontally. The same applies for every component and the more you add to the excavator the more expensive it becomes and more heavy-weight and so on.

My feeling when researching for this piece of text here.

Software has a special property, which basically means that copying it is very cheap. So one would say, that horizontal scaling comes for free? Well, there is still communication necessary. You can fork Bitcoin 100 times and then process 100 times the transcations, but it is no longer the same network. You might have a BTC on network #15 and someone else has a wallet on network #45, how do you interact with each other? You cannot. This is the big problem of horizontal scaling. In contrast for vertical scaling, we have the technological limits. Proof-of-Work and Longest-Chain-Rule can be scaled up by increasing block size and reducing block time. Seems to be very easy, but it is actually not an option. A fair competition and a real randomness of blocks being found is only guaranteed, if there is enough time to run the hunt for the next hash. The more you reduce the block time the more the network centralizes to those actors, which can communicate fastest to the others, the same applies with block size and bandwidth. Sad.

But what is the general approach to this? Do the cool blockchain kids of today try to scale vertically or horizontally? Turns out, both. The default solution for vertical scaling is Proof-of-Stake (PoS). The default solution for horizontal scaling is having multiple blockchains interoperating with each other. I will not explain all the fine details of PoS here, since I have done that in another article that is linked further above. We also don’t need to understand this stuff here, what we need to know is: PoS does not run a probabilistic puzzle, therefore the next block can be produced much faster. There is no need to have a long block time, the limit is basically how fast the members of the network can process a block and how fast they can communicate. Process time of a block is just a matter of CPUs or GPUs, something that can be scaled quite easily. Network delay is something that is ultimately limited by the speed of light, so a photon needs roughly 133 ms to travel around the earth. Signals in copper are 3 times slower, photons in fiber optics only lose 33% of their speed, so 200 ms is possible. However there are also routers in between and fibres might not be perfectly laid out and we need to have packets go back and forth, so let’s say 1s is reasonable.

It might be really hard to have a distributed application where all of it’s participants are informed of an update and agree to it in less than 1 second. It also depends on bandwidth compared to the block size, but for example Bitcoin blocks are 1 MB and this is not a bandwidth issue here. So if we assume Bitcoin block size and calculate the increase of throughput by running something that really has this 1s time, then we arrive at 600 times speed up. I will try to lay out how this is calculated and hopefully make it easy enough to understand. Bitcoin block time is 10 minutes and a minute has 60 seconds, so if we multiply that, we end up with 600 times the number of blocks in the same timespan. Nice speedup. Another complex calculation of Bitcoin’s 7 tx/s multiplied with 600 gives us 4200 tx/s, so in essence, Visa level is achievable by having a 1MB block every second. In reality though the timing is more between 4s and 7s, but we can also increase the block size as well, so vertical scaling looks good. Proof-of-Stake has another nice advantage, which is that it doesn’t increase global warming more than necessary. So we can get rich with it, but without destroying our planet. A nice addition on the list.

But here is one problem to mention and this has to do with this communication thing. In Nakamoto Consensus a new block can be broadcasted and you don’t need to wait for anything. The nodes communicate with a gossip network, so everyone tells its neighbors their new data (blocks). Why is there no need to wait for anyone? Because anybody can verify blocks without asking anyone. This is a very great feature and with PoS this is not possible. There is no cryptographic puzzle, so you cannot just verify a hash. You need to know if all the others have agreed. This means if there are 100 block producers, each of them needs to communicate with 99 others. If there are 10.000 block producers, each of them needs to coordinate with 9999. This basically means for N participants each one needs to communicate with N-1. So the communication scaling is N*(N-1), for large N the 1 does not matter, so we end up with N². We call this the N²-Problem and it means we can’t scale the block producers, or better say validators, indefinitely with PoS.

So is there a solution for the N²-Problem? Sure. A very common one is the validator delegator split. The blocks are produced by validators, who run nodes and propose blocks one after each other and vote if these blocks are valid. Their voting power is determined by their own stake plus the delegated stake from the delegators. The delegators do not run a server, they just bond their stake to a validator, who runs a server. The N²-Problem is solved, because there is only network communication necessary between the validators, so the number can be small, for example 100. What is the reason why we want to have as many nodes as possible? There are 2, one is prevention of cartel formation/collusion and the other is partition tolerance. So are we performing worse now? Well, the cartel formation or collusion part is actually still prevented by delegators. Since these can scale indefinitely, to form a cartel one needs to bribe the delegators stake as well. In case some cartel is forming, then the delegators should withdraw their stake from this cartel and redistribute it to other validators. For the partition tolerance the delegators do not help and partition tolerance is also lowered by another property of most PoS implementations. Based on practical Byzantine Fault Tolerance the liveness of such networks is different to Nakamoto Consensus. When too many nodes go down, then the network halts and cannot produce a new block. In contrast Nakamoto consensus does not really care. When half of the miners go down, then the next block will take longer — 20 minutes if half of the mining power is affected — but this will relax back to the targeted 10 minutes, if the network remains split. So here is a key take home message: PoS lowers partition tolerance.

It is sad to have less partition tolerance, but on the one hand events where half of the block producers go offline, are so severe that it might be good to stop and wait for a moment anyway and on the other hand we get a nice feature for this tradeoff and that is finality. It means that blocks become final at some point. So in Nakamoto Consensus it is in theory possible that any block might be reverted later by a concurrent fork of the blockchain. That is not the case if some variant of pBFT is implemented.

Nice, so we are happy? Well, PoS has some other problems, too. Namely Long-Range-Attack and Nothing-at-Stake, which do not exist in PoW or not to that extent. These problems need to be solved in order to deploy a functional cryptocurrency. Let’s have a short look into these problems. If we implement PoS naively, which means like Nakamoto Consensus, but swapping PoW with PoS, then it is possible for an attacker to create an alternative reality. At some point in the past, an attacker could fork off and create their own reality. For example when the attacker has 1% of the stake, every 100th block is produced by the attacker. In this block the attacker maliciously increases their stake until after this has happened often enough the attacker has enough stake to take over the network. Once taken over, the attacker can produce many blocks very fast. Of course the attacker cannot fool other validators, who don’t agree to this reality. But imagine someone joining the network and wanting to download the blockchain history. This newcomer cannot decide if the reality presented by our attacker is real or not. Even more so, if he also downloads a competing reality from another validator, then using the Longest-Chain-Rule would lead to the faked chain to be more trustworthy. Since this attack works better the earlier it fakes blocks, it can basically start from the genesis block forging a different reality. That is why this is called Long-Range-Attack. Why doesn’t this work in PoW? Because you need to have the mining power to forge many blocks with high difficulty. You cannot just make up a blockchain in PoW with very high difficulty and being very long without having a lot of mining power. So any outsider can easily see your alternative fork is not the longest (heaviest) chain. Contrarily to what I just said the name Long-Range-Attack originally comes from PoW but what it means for PoW is a bit different (if you want to know, you can read it on the ethereum blog).

Ok, so how can we solve this? Well, actually we can’t use Longest-Chain-Rule anyway, so we need to change some parts anyway. The solution can be checkpointing. This basically means we decide that there is no way to revert some blocks. There cannot be an alternate fork overtaking these blocks anymore. This block is final. We have heard of this feature already and it is called finality. We also see that for Bitcoin it is hard to have such a checkpoint, since there is no one to decide when to do it. So what does it imply? Well, at these final blocks there is a fixed state, which we can write down and now it is sufficient for any outsider to just download this fixed state. This also means an outsider does not need to download the whole blockchain history anymore. Since all effects of the history are condensed in the state at this point. This is also called pruning. For example in Ethereum, without pruning the blockchain would be 4 TB big and with it, it stays quite small, only a few hundred GB. This still sounds like very much, but it is a big difference when it comes to what a server can easily handle.

However we see another point here, why it is hard for Bitcoin to do that. There is no state in Bitcoin. In Bitcoin blocks do collect a number of transactions which result in UTXOs on Bitcoin addresses. To know the balance of an address one needs to sum up all blocks. Ok, so how do these PoS/BFT consensus systems allow checkpoints? Basically they are defined in the protocol. So it would have been possible for Bitcoin to define such checkpoints as well, but it would mean a lot of implications, that Satoshi Nakamoto most likely wanted to avoid. In PoS/BFT this comes for free, since these implications are already bought. The most important one is waiting for all participants until they responded or at least the majority has responded. You cannot just go and say every 500th Block is final or so in Bitcoin, because that would give the lucky one to find such a block too much power. At such a point you need to make sure that everyone agrees to this checkpoint, since the good reasons to allow for competing forks still hold. In PoS/BFT we don’t need to insert such a thing into the consensus mechanism, since every block is produced by a vote. When 2/3 vote a block valid it becomes a block. In pBFT it even becomes a final block. So using this approach to make PoS a reality, the solution to the Long-Range-Attack comes for free. Nice.

Those of of you who can’t relate, please think twice.

Leaves us with the Nothing-At-Stake problem and it describes that there is nothing at stake when it comes to fork choice. In PoW if there are concurrent forks, you have to decide which to follow, since you cannot mine all of them. If between these forks there is just 1 transaction different, the hash will be different and you can only mine on one fork. This makes fork choice necessary. In PoS your coins get duplicated for each fork, so you can “mine” or better say validate on every single fork. This is even something that is not just possible but incentivized. It makes sense for you to follow any fork and receive the block rewards. Not a very good prospect if there might be 100 forks soon and nobody wants to solve that forkin’ mess. Even though such an attack never happened it might be the fear of it happening being one of the main reasons why early PoS Coins failed. There are some other factors, maybe the high pre-mine that was hated back in these days not giving “proper decentralization” or the lack of seeing and implementing the advantages of PoS. Discussing that is an interesting alternative topic, but let’s come back to what we are looking at here. We have an attack vector and it should be solved. How do we disincentivize validators from following each possible fork? Well, by punishing them for doing so. So in order to get the desired behavior, the protocol defines a punishment. So forks can still be done, but you have to decide. All members of your fork, slash the coins (punishment) of the members of the other fork and vice versa. If someone follows both, then slashing will happen on both. Other misbehavior is also punished by slashing coins and that is double spending and being offline for too long. With this combination of BFT + PoS + Slashing we have a system that makes misbehavior costly. The very same applies for PoW + Longest-Chain-Rule (Nakamoto Consensus). If you follow the wrong fork there, you lose your invested work, which means paid electricity, which means money. In case you misbehave and attack the blockchain properly you might also destroy the value of the coin you are invested in.

In PoS this is even tighter connected, since you must own the coin and not the hardware which might lose a lot of value, when the coin becomes worthless (for ASICs more than for GPUs). Ok great, so is both the same? Not the same, both approaches achieve the same goal, but there is a very important difference. In PoS you need to lock up coins and get punished in case shit happens, in PoW you need to burn electricity and only get something for it, if you don’t misbehave. Both achieves security for the decentral network, but one does not destroy the planet by wasting a lot of energy, which is an advantage. Just imagine how nice that would be.

With PoS this can become a reality without the part, where the planet is destroyed.

Now that we understand why PoS makes a hell lot of sense, we also want to understand why it is much faster. In Nakamoto Consensus the actual processing of transactions is not very costly. The costly part is solving the cryptographic puzzle. To make it fair the time to solve the puzzle must be many orders longer than the propagation time of a block. If there is not enough time, then either the finder of the last block is highly advantaged (extreme case) or the most central nodes in the network have an advantage (more realistic case).

Well, this is not 100% true, since for example in Ethereum there are Uncles. Uncles are blocks that were found and are valid, but do not belong to the main line of the blockchain. So just like that strange uncle on a family gathering these are not really wanted in the first place, but belong to the family anyway. For Ethereum this means the Uncle gets included, if it is valid and for the finder of a block this means you can get it into the blockchain even if someone else was just 1 second faster than you. This allows for a drastic reduction in block time (15s for Ethereum). But still Ethereum is not significantly faster than Bitcoin. So let’s go back to PoS. There we don’t need to run such a long timing for the puzzle, since there is no puzzle, we only need the time to let everyone get to know from everyone else that the block is valid. Then we can go on producing the next block. There are different ways of selecting the next block producer, but it does not really make a difference. The important part is that on a planet like the earth and with the speed of light the whole process can be done in a second or two. The size of the block is mostly limited by bandwidth, so this might increase in the future. For Bitcoin this also holds true, but there are more adverse forces being against bigger blocks and the reason is mostly to keep decentralization at maximum.

Now that we have a basic understanding of the main approach to vertical scaling, PoS, let’s now look at horizontal scaling. There are 2 candidates and they are quite similar. The one is interoperability and the other one is sharding. Interoperability also belongs into the compatibility category, since it connects different blockchains together. So if Interoperability is solved we can make incompatible things compatible. But if we can do that, we can also create 100 forks of a given blockchain and let each fork talk to each other and connect the stuff. 100 forks process 100 times the number of blocks of a single fork, so why this scales is obvious. The big problem here is: How to have trustless Transactions between two blockchains? This is a good question and I will try to answer it, by giving a simple example how one can connect 2 blockchains without inventing any new technology. By the way, I find it very strange that in this blockchain world you find 1000 articles explaining how awesome and important interoperability is, but almost never it is expained how it actually works. Even if someone goes on a specific reddit and asks, please explain it to me, then users keep responding this “Yeah the IBC protocol makes it possible to have 2 blockchains securely talk to each other”, ok fine, but how?

Somehow all different interoperability memes are about how people talk too much about it, without understanding it.

Let’s assume we have Bob and Alice, where Bob has a Bitcoin and wants to trade his Bitcoin with Alice, who has an Ether. Both understand that exchanges like Mt.Gox do exist or say do exist only a limited amount of time and might go pop tomorrow and don’t want to put their valuable coins on an untrustworthy exchange. Not saying the exchange is a scam, but what can it do? In the end it is a central entity, that can always fall to individuals mistakes, bad architecture or just someone randomly dying in an airplane, being the only one with the private keys of the exchange’s wallet. So both of them really want to do their trade in a secure decentral manner. Now they have heard about such a great concept that is DEX (decentral exchange). But unfortunately this only works if they exchange shitcoin A with shitcoin B and both are from smart contracts of the same (Ethereum) blockchain. In the case of 2 different blockchains, this is not possible.

So they have another idea. There are a lot of friends of the two, that are willing to help them, some of them know each other, but most of them don’t. So let’s take 5 of them, that don’t know each other. Now they create a multi-sig wallet on Bitcoin and do the same on Ethereum. What is a multi-sig wallet? This is a wallet, where all of the 5 have to sign a transaction to send it into the blockchain. You can also have a multi-sig wallet, where only 3 of 5 need to sign, but for this example it does not really matter. So these 5 create two multi-sig wallets. Now Bob sends his bitcoin to the multi-sig wallet on the Bitcoin blockchain and Alice sends her Ether to the multi-sig wallet on the Ethereum blockchain. If both do it, we go to the next step, if not, then the 5 create a transaction that sends the Bitcoin back to Bob or the Ether back to Alice. Note that no one can single handedly run away with either the Bitcoin or the Ether here.

Now that both have transacted their coins, one member of the 5 creates a transaction, which sends the Bitcoin from the multi-sig wallet to Alice’s bitcoin address and another transaction that sends the Ether to Bob’s Ethereum adress. Now the other 4 members of the 5 see that both transactions were created just as planned and sign both of them. After that both coins are sent to the other one and they have successfuly made a decentral trade. Again nobody could have walked away with the coins single-handedly. This would only have been possible if all of the 5 colluded. Still there might be reasons for them to do so, for example a large transaction, where stealing the coins and dividing by 5 is more valuable than the reputation they lose in this process. That is why we maybe want 100 instead of 5. So basically we have solved interoperability now? Well, it might be a bit troublesome to find 5 or even 100 multi-sig wallet holders for every trade two blockchain users want to make. One approach could be to just set up a team of 100, that keeps doing this thing for everyone. Unfortunately this increases the security demands on these even more.

Is there some way to ensure that they act properly? Well, punishment is the key again. How about letting these 5 or 100 lock up some coins and if they misbehave, these coins get slashed. This makes it much more dangerous to try and start a collusion with others, since these might report such a behavior. However it might be hard to find 100, where each of them is willing to lock up a big amount of coins. But wait, really? We actually have some candidates that already did so and these are the validators of our PoS blockchain. It becomes interesting here. These validators are even running a server anyway and even more so handle transactions. So great, we just use the validators and problem solved? Unfortunately problems arise again. One problem might be forks. Again. So even if these 5 or 100 see that both have sent their coins, then it might still happen, that in 20 minutes there is another fork of the blockchain taking over and in that Alice maybe did not provide their part of the deal. But if the other part of the deal was already sent, then there is no way to stop it. So actually one needs to wait until there will be no more forks possible. In theory this is not possible for blockchains like Bitcoin. What we need here is finality. Aha. So PoS also provides a nice feature for interoperability as well.+

The faster finality is the faster a cross blockchain transaction can be finally settled. But does that mean, it is not possible at all to include Bitcoin into our whole interoperable world? Well, actually it is possible to create something that is called a Peg-Zone or Bridge, which takes Bitcoin and creates pegged or wrapped Bitcoin for that. The pegged Bitcoin is now rooted on a PoS blockchain and has finality and the real Bitcoin is waiting in a wallet controlled by the Peg-Zone. When someone wants to have the real thing back, the minted/pegged Bitcoin is burned again and the real Bitcoin is sent to a Bitcoin address. In this case the Peg-Zone can never be 100% sure, that there will not be a competing fork taking over, but this is basically the same problem exchanges have today, when they accept coins of blockchains without finality. So this means in a very strict theoretical thinking, the state of the chain is never final, but after many blocks, for example 10, it becomes extremely costly to try an attack that is successful. It depends on many parameters, but the basic thing here is, that you need to be faster than the miners of the blockchain to forge an alternate reality. So you need to invest into more miners that already exist and then attacking might crash the price of that coin, rendering your attack unlucrative. If you have less miners, for example only 10% of the total mining power, then you can still try your statistical luck, since you can still be succesful in finding a competing block, where you steal the coins from the Peg-Zone, but doing this for 2 blocks becomes less likely. For example if you have a 10% chance to find the next block, then you only have a 1% chance to find the block after that as well and if you need to find 10 blocks to do the attack, you have a 0,00000001% of succeeding. In order to try that you still need to run these miners, which might cost some billions of dollars. It is unlikely that you can easily steal that much money, so in essence and for practical reasons, it is not possible to attack this Peg-Zone without losing money.

Those of you who can’t rela… wait we had this caption already.

Unfortunately this implies we have to wait for several blocks. In contrast to both sides having PoS with instant finality, we can do the thing with 1 block. Ok nice, is there something else? Yes. If we have these PoS chains set up, then it might be even possible to have shared security, which means we have shared validators on both blockchains. These are very interesting for our example, because they can be punished on both sides if they misbehave. Making bribing more complicated thus increasing security. Ok nice, so this is everything I need to know? No. There is even more. Basically we don’t need to send coins. We can also just send transactions with information across different blockchains. This means smart contracts can talk to each other. This is also a nice feature. Ok, but now we have everything? Well, let’s say we understand the basics and there are many details, for example why it makes sense to have a light client of each blockchain, allowing the members of the other blockchain to easily verify things. But let’s leave it at that for now.

And let’s go to sharding, which is a bit more accessible now that we have read so much and understand a lot more about how all of this stuff interacts. Sharding scales for the same reason interoperability scales, but instead of having multiple independent blockchains in parallel and make them communicate, we want to find a way to split up a single blockchain into multiple parts, or say shards. Sounds like it would end up with the same result, but it doesn’t. The source of the differences is that the first approach copies the blockchain and gives each copy its own infrastructure and the second approach has the same infrastructure trying to split up its workers over many parts. This is why for interoperable blockchains the big question is how to make these disjoint parts interact safely with each other and for shards the question is how to organize all these workers so that single entities cannot corrupt the whole thing.

After we see this approach as something where the workers of the infrastructure are divided and no worker does all of the work, we get an idea why sharding is good for scaling. The work is producing blocks and the workers are the validators. Sharding allows to split up this work, so that only a small fraction of the validators work on the same part of the network. The reason why Nakamoto Consensus does not scale up with more nodes is that every node has to do all the work. If we split up the work over many nodes, then having more nodes means we can split up the work in more pieces, making every piece smaller and thus enabling scaling. So the idea is that an incoming transaction goes to a shard and is processed there. It gets included in the block on this shard and the other shards do not really need to bother about this transaction. But wait, how does this work, if the transaction transfers some coins from an address that is not on this shard? Well, this indeed does not work. There are many possible ways to organize these shards, but luckily this is not a totally new topic for blockchains.

There is massive parallel computing, where many computers solve a heavy computational task together, for example to find signals of extraterrestial intelligence in the radio noise that comes from space (SETI). The other field is databases especially with clustered file systems and massive data storage. The different parts of huge databases, are often called shards, so this is where the naming comes from. In these fields smart researchers have already racked their brains over these problems and the solutions differ depending on what exactly your system should be good at. If you write very often in the database and read only rarely, it makes sense to organize the system so that writing speed maximizes. This means incoming data from a single source often ends up on totally different shards, but that is what allows very fast writing. If you have a lot of requests and these requests represent some kind of condensation of data, it might be a lot better to put data that belongs together on the same shard. For example if you have a database, which stores all data of people living on this planet and requests are something like “give me the average age of people living in all cities”. If the people’s data is distributed according to their living place on shards, then each shard can calculate the data for its cities by itself and then just report back the results for each city. If the data was lying around everywhere, then each shards needs to share the data with each other shard and then the calculation can be done, which is more work.

It even gets more complicated, for example if we want to correlate this information with some other information, like how tall these people are. Then we can’t just sum up their ages and send the sum together with the number of people sampled to the next shard and so on. Then we might need to collect all of that data at some place first and then calculate the complex correlations. If we look at massive parallel computation it is the same story. Basically Bitcoin mining is a huge massive parallel computation operation and synchronizing it is very easy. The reason why this is so easy, is because all of the work does not matter at all, except for the one attempt, that finally found the block. It is somewhat similar for protein folding computation, which is also an example for such a distributed computation task. There the most important result is the one randomly generated parameter set that gives the most stable protein. For SETI the recorded noise can also be distributed quite well on different computers, but there is also the question of frequency involved. If a signal has a wavelength so long that the signal is sent over a duration of 10 years, then it might not be possible to find it, if the recorded noise is split into 100 second long pieces. Of course there are many clever things to be done, but that is not the aim of this article here :D

So how about sharding in blockchains? Well, there is obviously data storage involved and there is computation involved, great. So we have the worst of both worlds. And as luck has it, there is another big pile to be shit on top of that and that is secure computation. In the other examples there is no single entity that acts maliciously or even might be incentivized to do that. So we have data storage, computation and security. All of these must be nice. But we are lucky and we know quite a lot about the data to be stored and the computation to be done. For example for a transaction of coins, one must only check if there are enough coins available for the sender, the receiver is always fine. And the computation of smart contracts can work so that all the data is saved in the smart contract. So whenever someone sends a transaction to a smart contract, then all the data is in one place and not scatterd everywhere. Well, not necessarily all of the data, since a smart contract can also access the data of other smart contracts, but in most cases smart contracts care mostly about it’s own history and transactions sent to it. In addition we know that blockchains are not write heavy tasks. Since writing costs some fee and even though sharded blockchains aim for handling 1000 times more transactions than classical systems, it is still not a database writing GB/s of data. An examples for such a data storage si the CERN Large Hadron Collider experiment.

Looking deep into these eyes, you can see the reflexion of prices moving up

Still we can see now that there are 2 bottlenecks and one important constraint. The bottlenecks are computation and data storage, the constrain is everything must be secure. Taking this into account we end up with an architecture that does not randomly write wherever possible, but rather organizes data together into the same shards that belongs to the same smart contract and addresses interacting with each other and these smart contracts. Ok nice, so we have many shards and the addresses and smart contracts are split up over these shards. For each shard we assign some nodes that propose and validate new blocks. Furthermore when there is cross-shard communication necessary, these nodes send the data around. Sounds fine and not too hard? Well, if we have a fraction of the nodes on each shard, then it is much easier for these nodes to form a cartel or bribe some other nodes to do nasty things. They could just start forging malicious blocks and send this information to other shards, corrupting the whole blockchain. That would be quite bad.

Luckily there is a solution for this and we just shuffle these nodes constantly. If these nodes are exchanged and presented new partners every few seconds, it becomes very hard to collude or bribe others. Since you need some coordination and time to do that, it becomes very hard if constantly new players show up on your shard and you have to move to another shard quite often. In the end you have to collude with everyone and this means that it is as secure as the good old blockchains. Random shuffling also solves another problem, namely that the coordinator of all those shards could also act maliciously by dropping some cross-shard transactions or forging fake cross-shard transactions. By moving around these validators can easily recognize misbehavior of a coordinator node. So everything is fine now with sharding? No. Shuffling has a very delicate implication. It basically means that we have to send all the data of the shard we are leaving to someone else and we need to receive all the data of our new shard. This means a lot of bandwidth.

Alternatively we could store a lot of data, for example of all shards, but then there is no advantage in sharding. If every node has to store everything, there might only be a scaling of computation power, but not of data storage. A single blockchain today is already 100s of GB big, so storing 64 shards or more, might not be that easy and drive out many nodes, leading to centralization, something we don’t want. If we give up decentralization, we don’t need to run a blockchain. Then we can stick to the good old databases. So we either need a lot of data storage or we need a lot of data bandwidth or we need to have nodes stick to the same shards, which leads to inferior security. This is called the Data-Availability-Problem. It is nice to note that this is a trilemma, so instead of an ordinary dilemma, where two options are presented and both are bad, we have now three options, and all of them are bad. Unfortunately for the trilemma, there is a genius solution for this problem.

What is the solution? We can split up the nodes, which have the data and the nodes that validate the blocks. But how does that solve the problem? It is just more data transfer necessary because nodes have been split? Well, the trick is that the data nodes stick to their shard and the validating nodes are randomly shuffled. This means that the data nodes hold all the data of their shard and don’t have to reload new data of other shards all the time. These nodes provide the transactions to form blocks or even propose the blocks, this depends on the specific implementation. But whatever is the case, the validators just take these transactions and forge a block or just check the proposed block. If the validators agree, the block is published. With this approach the validators can be shuffled quite often, so that it is hard to collude. But it is not necessary to transfer huge loads of data all the time. Great, so we have solved sharding? Well, there is still a lot to take care of, but in essence it is possible. We will look at some other finer problems, when we see how it is implemented.