Shadowing Solidity Storage Variables in Memory
By: Jules Goddard
Tags: Solidity, Testing, Ethereum, Sorting Algorithms, Blockchain Technology

This article discusses the technique of shadowing Solidity storage variables using memory in Solidity, the defacto smart contract language for the Ethereum blockchain.

Sort those jelly beans, quickly. Photo by Patrick Fore on Unsplash Retouched by Kitty West

Background

During the development and testing of Datona Labs’ Solidity Smart-Data-Access-Contract (S-DAC) templates, we needed to sort some small sets of persistent data, for example:

import "DiSort.sol"; contract SDAC100 is ... { uint[] fileKeys; function sortFileKeys() internal {

DiSort.quickSort(fileKeys);

} // ...

Solidity effectively has 2 data locations, storage and memory. Storage is persistent across multiple smart contract calls, whereas memory is transient and only exists for each call. There is a 3rd data location, known as calldata, which is effectively the same as memory.

The array in the code fragment above must be persistent across multiple contract calls, hence it must reside in storage.

But access to storage consumes much more gas than memory, so any algorithm which consistently accesses the data is going to be very expensive.

Any of the traditional sorting algorithms, such as Insertion_Sort or Quick_Sort, uses a lot of data accesses in order to carry out element comparison and swapping. A non-comparison sort, such as Counting_Sort saves on element comparison and swapping, but only works with a known small range of numbers. See this article by the same author which discusses the possible benefits of non-comparison sorting in Solidity, and provides the source code for the sorting algorithms. It also considers the use of different array data types.

The sorting algorithms must be rewritten in order to address the correct data location, for instance:

function quickSort(uint[] storage data) internal { // ...

Shadowing Data

One technique to reduce gas consumption is to copy the persistent storage array to memory (shadowing the original data), perform the sort, then copy the array back again.

function sortFileKeys() internal {

uint[] memory shadow = fileKeys;

DiSort.quickSort(shadow);

fileKeys = shadow;

}

This function consumes gas in order to copy the array between data locations, but will that be insignificant compared with the cost of sorting?

Let’s measure some example data to find out.

We will measure different sorting algorithms across our expected number of elements, in both storage and shadow memory data locations. The element values will be generated using a pseudo-random number generator which produces a deterministic sequence of values.

Results

All the contracts were run and the data collated in spreadsheets to produce the following results.

(Fig 1) Gas cost of different sorting algorithms in Storage

Note that Counting Sort is much faster then Quick Sort and Insertion Sort in storage (Fig 1).

(Fig 2) Gas cost of different sorting algorithms in shadow Memory

Note that the cost of copying from storage to memory (S => M) and back again (M => S) dominates the consumption of gas (Fig 2).

(Fig 3) Gas cost of different sorting algorithms in Storage cf Memory

The combined curves are shown for comparison (Fig 3).

The outstanding result is that the Counting Sort is almost as fast in storage as in shadow memory (Fig 3). That’s because it is skipping all the expensive element comparison and swapping. Unfortunately, Counting Sort is only suitable where the element values are in a small known range.

Also outstanding is that the gas consumption of the comparison sorts (Quick Sort and Insertion Sort here) using shadow memory is 6 times less than using storage (Fig 3), despite the additional gas consumption due to copying the array between different data locations (curves S => M and M => S in Fig 2).

As an aside, if the data is likely to be in a sorted state before invoking the sort routine, it would be beneficial to only copy the array back from memory to storage if the data has been changed.

function sortFileKeys() internal {

uint[] memory shadow = fileKeys;

if (DiSort.quickSort(shadow)) // if data has changed

fileKeys = shadow;

}

The sorting routine would require modification to indicate whether the data has been changed in this case.

Conclusions

Using sorting is painfully expensive in Solidity’s storage data location, so it is always worth shadowing the data in the memory data location and performing the sort there.

Other algorithms that perform many data accesses may also benefit from using the shadow technique.

As for sorting algorithm, use either Quick Sort or Insertion Sort. The latter is a much more simple algorithm and thus produces less Bytecode. It gets our vote.

Bio

Jules Goddard is Co-founder of Datona Labs, who provide smart contracts to protect your digital information from abuse.

website — twitter — discord