PoCo Series #9 — Indexing smart contract activity. It’s harder than you think!
By: Hadrien Croubois
Tags: Tech, Smart Contracts, Ethereum, Blockchain, Poco

In the previous articles of this series, we extensively discussed iExec’s use of smart contracts to enforce the PoCo protocol in a trustless and decentralized manner. Their design impact the entire platform, as they set the rules on what can and cannot be done, by who, and which impact these actions have on the participants’ balance and reputation.

Previously, the series discussed the latest developments that will improve the modularity and future-proofing of the iExec platform. In this article, we will cover the process of handling data by smart contracts, and the indexation of smart contracts’ activity.

For an app to work well, and to integrate within a hybrid on-chain/off-chain ecosystem, it is not enough to just implement your logic into smart contracts. You also need to understand how these smart contract store data and trigger events.

Smart Contract Auditability

Smart contracts are often presented as “auditable by design”. What this usually means is that both the code, and all transactions are publicly available, meaning anyone can observe the state of the contract, and understand what everybody else has been doing. While this is technically true, the code and transactions can be difficult to follow and process. Contracts may include accessors that give read-access to some of the data, but even if there is no accessor, it is still possible to read anything that is part of a smart contract’s storage. This explains why a smart contract cannot securely hold a private key or produce a cryptographic signature. To access the state of a smart contract you can either look it up in an ethereum node or rebuild it through the process of synchronizing your own node, using the history of all past transactions.

However, the state of a smart contract is not everything. Past states (the balance of <xxx> at block height <yyy>) and past events (Transfer from <xxx> to <yyy> of value <zzz> in transaction <www>) also provides crucial information about a smart contract’s activity. Storage, memory, calldata, logdata … they are many memory spaces that smart contracts have read/write access to. Before going into the many challenges of smart contract indexation we first have to go into the mechanisms that power the EVM.

How is data stored on-chain, how can I access it?

People often think of the blockchain as a simple distributed ledger, composed of blocks that people can submit transactions to. However, an ethereum (EVM) blockchain is actually a lot more than this. First, we have to differentiate the following:

Calldata is the name given to the transaction’s data field. It is mostly used when deploying new smart contracts or submitting transactions to existing smart contracts. It is the input of smart contract calls.

is the name given to the transaction’s data field. It is mostly used when deploying new smart contracts or submitting transactions to existing smart contracts. It is the of smart contract calls. Memory is the name given to all the temporary variables used during the execution of smart contracts. It is only used during the resolution of a call, but it is not stored or accessible after the call is complete.

is the name given to all the used during the execution of smart contracts. It is only used during the resolution of a call, but it is not stored or accessible after the call is complete. Storage is the name of the long-term memory space that each smart contracts can read from and write to. This is the most expensive (in terms of gas) memory space to interact with. This contains all the state/variables that smart contracts have to keep between transactions. A good example would be ERC20’s balances.

is the name of the memory space that each smart contracts can read from and write to. This is the most expensive (in terms of gas) memory space to interact with. This contains all the state/variables that smart contracts have to keep between transactions. A good example would be ERC20’s balances. Logdata is the name given to the memory space where event logs are stored.

Calldata and logdata are included in the blocks. This guarantees its long term availability. On the other hand, they cannot be read/overwritten by latter transactions, which means they can be stored on slow/cheap devices such as hard drives. Storage, on the other hand, can be read from and written to by future transactions. This means nodes have to keep it in fast (RAM/SSD) storage, explaining the extensive cost of writing to storage, and the refund one can get when cleaning it up.

Much more could be said about the cost of storage, including the difference between clean and dirty storage slots. I would encourage any (aspiring) smart contracts developers out there to learn about this, as well as the security consequences of gas cost changes.

What are the challenges?

As previously mentioned, storage is very expensive, which means that it should only be used for values that either has to be accessed by further calls or should be easily accessible through accessors. Putting everything in storage is however a bad (and very expensive) option.

To come back to the example of ERC20 tokens: here, you want the balances to be in storage but not the history of all transfers.

Transfers are represented as events (in logdata). They are much less expensive but are more difficult to retrieve. You need to know which transaction/block to look for, and if they are old nodes might not hold them anymore (A full node is required).

Past states are even more difficult to retrieve. Old blocks transactions (calldata) and events (logdata) are not enough, you need to know the storage state as it was at a certain block height. This is only stored by archive nodes, which are expensive to run due to the large amount of storage they need.

Rather than constantly looking for past events or past state, many dApps prefer building ad-hoc, centralized, databases that are fed from on-chain activity and, in turn, provide structured data in front-ends. The underlying smart contracts are still trustless, but auditing them requires either very technical skills or trusting a centralized source.

iExec is using this pattern to feed frontends like the iExec Explorer and the marketplace.