Sorting in Solidity without Comparison
By: Jules Goddard
Tags: Ethereum, Sorting Algorithms, Testing, Solidity, Blockchain Technology

Sorting in Solidity without Comparison

This article discusses the cost of sorting small sets of numbers in Solidity, the defacto smart contract language for the Ethereum blockchain.

Background

During the development and testing of Datona Labs’ Solidity Smart-Data-Access-Contract (S-DAC) templates, we wanted to sort some small sets of small numbers, for example:

function sortHands(Hand[noPlayers] memory hands) internal pure {

for (uint player = 0; player < noPlayers; player++) {

sort(hands[player].cards);

}

}

Of course we want to use as little gas (few cycles) as possible because blockchain languages like Solidity are very expensive to run compared with normal systems and the gas actually costs a measurable amount of money.

A serious test. Photo by Inês Ferreira on Unsplash

Sorting Options

Sorting Algorithms

There are many possible sorting algorithms to consider, for instance see Merge_Sort or Quick_Sort, of which we have only considered the following for this article:

Sort Name Performance Quick Sort O(n log n)

Insertion Sort O(n²)

Counting Sort O(n + setSize)

Unique Counting Sort O(< n + setSize)

We presumed that the complexities of Quick Sort would be overkill for sorting a few small integers but useful for comparison purposes.

Insertion Sort is well known — the code is provided below.

Counting Sort uses the values of the data itself to drive the algorithm, and thus performs no comparisons. Unique Counting Sort further refines this by assuming that the numbers in the small set are unique.

Array Data Types

In addition to the range of sorting algorithms, implementation in Solidity includes the following possible array data types for holding small numbers:

Declaration Description bytesX a fixed packed byte value type byte[] a dynamic byte array

uint8[] a dynamic uint8 array

uint[] a dynamic uint array

bytes a dynamic packed byte array byte[X] a fixed size byte array

uint8[X] a fixed size uint8 array

uint[X] a fixed size uint array

In the bytesX value type, individual bytes have to be extracted and inserted using bit-wise operations, so we decided that it was impractical and we did not consider it further. For more details, see Solidity_Fixed_Size_Arrays.

We would prefer to use dynamic arrays if possible, because of their general applicability elsewhere.

Array types use a 32 byte word for each element, so both byte and uint8 arrays wastes most (31/32) of the space it consumes. See Solidity_Arrays.

For that reason, it is also worth considering the bytes packed array.

Unusually, the declaration of a bytes array is without the [square brackets] normally associated with an array declaration, even though it is treated in other respects as an array with indices and length. This syntax means that it is not possible to declare a fixed bytes array.

The bytes array type packs the bytes into 32 byte words which means that it will use a lot less memory than other array types, but also that indexed operations will be slower than an unpacked array due to the additional packing and unpacking code. For small temporary data sets, that memory usage is probably not significant. However, for data held in storage, Ethereum’s persistent blockchain data area, it is very significant and would weight heavily in any decision concerning data type selection.

Other Factors

In addition to testing different sorting algorithms, and different array data types, we would like to consider different numbers of elements. For these tests, we will use 10, 13 and 17 out of a set size of 52.

Lastly, we would also like to compare the gas usage of optimised and non-optimised code.

There is a tradeoff between the sorting algorithm, the array data type and the size of the data. The rest of this article describes testing all the options and identifying the ideal combination for our purposes.

Measurements Performed

The sorting algorithms were coded in Solidity along these lines:

function quick(<<type> memory data) internal pure {

if (data.length > 1) {

quickPart(data, 0, data.length - 1);

}

} function quickPart(<<type>> memory data, uint low, uint high) internal pure {

if (low < high) {

uint pivotVal = data[(low + high) / 2];



uint low1 = low;

uint high1 = high;

for (;;) {

while (data[low1] < pivotVal) low1++;

while (data[high1] > pivotVal) high1--;

if (low1 >= high1) break;

(data[low1], data[high1]) = (data[high1], data[low1]);

low1++;

high1--;

}

if (low < high1) quickPart(data, low, high1);

high1++;

if (high1 < high) quickPart(data, high1, high);

}

} // - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - function insertion(<<type>> memory data) internal pure {

uint length = data.length;

for (uint i = 1; i < length; i++) {

uint key = data[i];

uint j = i - 1;

while ((int(j) >= 0) && (data[j] > key)) {

data[j + 1] = data[j];

j--;

}

data[j + 1] = key;

}

} // - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - function counting(<<type>> memory data, uint setSize) internal pure {

uint length = data.length;

uint[] memory set = new uint[](setSize);

for (uint i = 0; i < length; i++) {

set[data[i]]++;

}

uint n = 0;

for (uint i = 0; i < setSize; i++) {

while (set[i]-- > 0) {

data[n] = i;

if (++n >= length) break;

}

}

} // - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - function unique(<<type>> memory data, uint setSize) internal pure {

uint length = data.length;

bool[] memory set = new bool[](setSize);

for (uint i = 0; i < length; i++) {

set[data[i]] = true;

}

uint n = 0;

for (uint i = 0; i < setSize; i++) {

if (set[i]) {

data[n] = i;

if (++n >= length) break;

}

}

}

Note that Quick Sort is by far the most complex with multiple loops and recursion.

Also, the Unique Sort is simply a slight optimisation of the Counting Sort.

The <<type>> in the code fragments above depends on the data type used, such as uint[] as described above in Array Data Types. Sometimes other minor editing of the functions was required, for example casting a uint to a byte.

You may have noticed that these functions are all classified as internal. This means that they must be available to be used at runtime. The source code can be added to our application contract or placed in a base (inherited) contract or in a library as we do here. In any case, the code will be included into our application’s byte code.

If the functions are placed in a library and classified as external then, despite the same syntax, the array data will be copied from memory to the Ethereum Virtual Machine (EVM) calldata area whenever the functions are invoked at runtime. Obviously this is slow, but even worse, array parameters are not copied back again when the function returns. The practical solution to this is for the functions to return the manipulated arrays. There may be some occasions when it is viable to use the very expensive Ethereum storage data area.

Measurement Technique

The tests were performed using Solidity compiler v0.6.8 (Byzantium).

Each of the sorting algorithms (above) were written in Solidity to an array data type specific library file, such as DiSortByte.sol:

// DiSortByte.sol // provide sort of byte[] pragma solidity >=0.6.0 <0.7.0; library DiSortByte { function sortInsertion(byte[] memory data) internal pure {

// as above

}

}

This library was imported into the array data type specific test files:

// TiSortByte.sol // test sort of byte[] pragma solidity >=0.6.0 <0.7.0; // libraries

import "DiSortByte.sol";

Each of the sorting algorithms, Quick, Inspection, Counting, Unique and a base (overhead-measuring) function, was tested in this manner:

function testSortInsertion() internal pure returns (string memory) {

uint length = 13; // initialise test data

byte[] memory actual = new uint[](length);

fill(actual, actualData()); // function under test

DiSortByte.sortInsertion(actual); // check test data

byte[] memory expected = new uint[](length);

fill(expected, expectedData());

return check(actual, expected);

} function gasCostSortInsertion() public returns (string memory) {

return gasCost("testSortInsertion ", testSortInsertion);

}

The length was set to 10, 13 or 17 according to the test.

The actual and expected arrays were allocated according to the array data type under test as follows:

// dynamic data types

byte[] memory data = new byte[](length);

uint[] memory data = new uint[](length);

uint8[] memory data = new uint8[](length);

bytes memory data = new bytes(length); // fixed data types

byte[13] memory data; // array length set veto 10, 13 or 17

uint[13] memory data; // "

uint8[13] memory data; // "

The fill and check functions were amended appropriately for each different data type. The test data was:

// 10 elements

input: 44, 8, 24, 39, 3, 18, 34, 50, 13, 29 output: 3, 8, 13, 18, 24, 29, 34, 39, 44, 50 // 13 elements

input: 46, 10, 26, 42, 6, 22, 38, 2, 18, 34,

50, 14, 30 output: 2, 6, 10, 14, 18, 22, 26, 30, 34, 38,

42, 46, 50 // 17 elements

input: 46, 16, 37, 7, 28, 50, 19, 40, 10, 31,

1, 22, 43, 13, 34, 4, 25 output: 1, 4, 7, 10, 13, 16, 19, 22, 25, 28,

31, 34, 37, 40, 43, 46, 50

The gasCost function (see Gas_Cost_Of_Solidity_Functions) measures the gas cost of the supplied function and conveniently appends the result of the test to a report string which is maintained in storage and can be viewed as required. Typical output is:

uint[](13) non-optimised

sort base GasCost: 7728

sort quick GasCost: 19191

sort insertion GasCost: 22080

sort counting GasCost: 22130

sort unique GasCost: 18839

The base value is the gas cost of using the test harness, so that figure was subtracted from each of the subsequent figures in the results spreadsheet.

Results

All the contracts were run and the data collated in spreadsheets to produce the following results.

Sorting Algorithm, Array Data Type and Optimisation

Here are the collated figures for each of the sort algorithms we employed, for each data type we tested, both non-optimised and optimised:

Measuring gas cost of different Sorting Algorithms, Array Data Types and Optimisation

The outstanding result is that the Unique Counting Sort is significantly faster than the Quick Sort and Insertion Sort and, rather surprisingly, even the Counting Sort from which it is trivially derived.

Other takeaways are that the fastest dynamic array type (the first four sets of columns) by far is uint[]; that although the fastest array data type is a fixed size array uint[13], the performance of the far more generic dynamic uint[] is so close that it probably makes the specialisation unnecessary; that optimisation actually has negative results on some of the more esoteric data types (byte[] and bytes).

Number of Array Elements

Having determined that the most sensible array type is uint[], we performed some more tests on different array lengths, namely 10, 13 and 17 elements.

Measuring gas cost of different Sorting Algorithms and array lengths

Again, the Unique Counting Sort generally produced the best performance for this test data, and the Insertion Sort is demonstrating its O(n²) performance curve.

As a further measure of the affect of the number of elements on the algorithm, we generated some additional tests using a number of elements with random values. These don’t claim to be true random numbers as per, for instance, this article. Instead, the random values are deterministic, and the output sequence is dependent on the seed value. This is ideal in a test environment.

We can seed our random number generator with something like the Ethereum block time. But for these tests, the default seed value was used for each test in order to ensure deterministic behaviour.

The resultant behaviour on the random numbers generated was:

Measuring gas cost of different Sorting Algorithms with different lengths of random values

Some trends are plain to see. Quick Sort and Insertion Sort are fastest for 10 elements or fewer, and Unique Sort is fastest for greater numbers of elements. Insertion Sort is rarely as cheap as Quick Sort and soon ramps up its gas consumption. The Unique Sort cost tracks the Counting Sort cost and is always cheaper. Both these sorts have linear progression in gas consumption.

Conclusions

Using sorting is painfully expensive in Solidity, so it is worth investigating which method works best for your specific data.

When using Ethereum memory, since all the array data types (except bytes) consume the same amount of memory for the same number of elements, it is not worth considering any other array data type than uint[]. Indeed, it is probably rarely worth considering using any other type than uint for any algorithm.

When using Ethereum storage, it is worth considering bytes instead of uint[] because of the reduction in use of this very expensive resource.

We note that Solidity’s optimisation option does not cope well with byte[] and bytes, making its use very difficult on contracts with mixed data types.

When sorting less than 10 numbers, use Insertion Sort.

When sorting more than 10 numbers from a small set, use Counting Sort. If they are unique, use Unique Counting Sort.

Otherwise use Quick Sort.

In our particular case, the best solution is using a Unique Counting Sort on a dynamic uint[] array.

Bio

Jules Goddard is Co-founder of Datona Labs, who provide smart contracts to protect your digital information from abuse.

website — twitter — discord