A data structure for sorting in linear time
By: Victor-Teodor Stoian
Tags: Data Structures, Algorithms, Computer Science, Optimization

Initially, the article was supposed to have a more artistic title, “A tale of empirical mediocracy from theoretical supremacy”, but it would have been too abstract and maybe even pushed the reader to make false assumptions about the content, which is a commentary on how a nice idea behaves in practice.

The idea for this started as a curiosity. I was curious to see if I can optimize a dear algorithm of mine, Insertion Sort. For those that are not familiar with the algorithm or just need to have their memory refreshed, Insertion Sort is a sorting algorithm that works by computing a sorted partial array from the initial one, by finding the correct position for each element.

Visual description of how Insertion Sort works, credit to: https://www.hackerearth.com/ for the image

As it can be clearly observed, the worst case for this trivial algorithm is O(N²). A trivial optimization is to use binary search to find the correct position for an element in O(log(N)) worst-case time complexity. But the problem is that after finding the correct position, all elements need to be shifted again to make space for the new one, which is again O(N), so the total complexity is still O(N²), but it moves a bit faster.

I was curious to see if I can use some data structure that allows me to use binary search on the partial solution but to make an insertion in O(log(N)) in the worst case. This could have been achieved by using a Skip List, but I wanted to see if I can do it even faster and soon I realized that what I was in fact trying to achieve, is to come up with a data structure that would allow constant-time insertion, constant-time lookup and would keep the data sorted. Sure, this would mean that the total complexity of sorting an array would be O(N) and this is impossible to achieve by using a comparison based algorithm, as seen here: https://www.cs.cmu.edu/~avrim/451f11/lectures/lect0913.pdf

I was thinking of ways to exploit the natural ordering of the input elements, thinking whether there is some mathematical solution for my wish or a clever trick, but I realized that I can use a Trie, or Suffix Trie, in the same way as using a frequency array, but with the benefit of having all the operations in constant time with regard to the number of elements.

But what is a Trie?

Simply put, a basic Trie is a data structure in the shape of a tree, where keys are represented by the path from the root, which is the empty set, to the corresponding element.

Example of a canonical Trie, credit to: https://en.wikipedia.org/wiki/Trie

For simplicity, I will deviate from the canonical Trie and present the modifications as well as the methods that my version of the data structure supports. Although we can use a Trie to also sort words in lexicographic order, I will present here the simplified version which sorts just numbers. Therefore the edges from any node to another can contain any digit between 0 and 9, this being our vocabulary.

So we can view all numbers as keys and use a Trie to store how many times if any, that number appears. Because at each level, the node has all the possibilities for the next key in sorted order, it means that a preorder traversal of the Trie will return the sorted array of elements. Even more, by storing the number of elements that will end on the level immediately below this one, which I called the “horizon” of each node, we can answer the question “After an arbitrary number of insertions, what element is on position k in the sorted array?” without having to sort the whole array and in fact, as we will see, in constant asymptotic time O(1) with regards to the number of elements.

As an example, the horizon of the root in the photo above is 1, because there is only one number composed of just a letter (“A”). The horizon of “t” is 1 and the horizon of “te” is 2. This notion will be useful when retrieving the number on a certain index.

The algorithm supports the following methods:

-Insert(numberToInsert) → inserts the specified number in the Trie and preserves the “sortedness” of the array

-ReturnOrdered() → returns the whole sorted array

-GetElement(index) → return the input element that is on the position ‘index’ in the sorted array.

Theoretical Analysis

We need to make two assumptions: the size of the alphabet of the input elements, let’s name it V, which denotes of characters that form the input elements, will remain constant and that the maximum size of the input elements is bounded by K. Using these two assumptions, we can easily see that the insertion and lookup processes have constant asymptotic time O(1) with respect to the number of input elements N and that creating the Trie and returning the sorted list of input elements has the asymptotic complexity O(N).

For insertion, the algorithm needs to go from the root of the Trie to the corresponding node which can be a leaf or inside the structure. The maximum number of operations that are done for insertion is O(K) where K is the maximum length of a key, but we assumed the length is bounded, therefore K is constant and the complexity becomes O(1).

The analysis of the lookup procedure is a bit more complicated. It uses the horizon to figure out if the index we are looking for is the immediate child of the current node. If it is not, we just go to the next node. This means that in the worst case (when the index corresponds to the greatest value in our input array) it will go through all the nodes in the Trie until it finds the correct one. The maximum depth of the trie is O(K) and each of the nodes will have V children. Therefore, the worst case will be for a full Trie (we can use lazy update to minimize the number of nodes), where the index corresponds to the greatest element and it has the complexity O(V^K), but as V and K are constants, the final complexity is O(1).

Empiric results

I have compared my python implementation of this algorithm with my own implementations of MergeSort on different tests.

The following are the results on complete, ascending input spaces (the input contains all the elements from 0 to N) for N going from 100 to 10 000 000:

We can clearly see that all times grow linearly.

The following are the results on complete, descending input spaces (the input contains all the elements from N to 0) for N going from 100 to 10 000 000:

The following are the results on sparse, ascending input spaces (the input contains elements from 0 to 1000 which are repeating) for N going from 1000 to 10 000 000:

The following are the results on sparse, descending input spaces (the input contains elements from 1000 to 0 which are repeating) for N going from 1000 to 10 000 000:

As expected, the Trie behaves better on inputs that contain few unique numbers than MergeSort and in all cases, the time taken for inserting a new element and looking up the element on a certain position (all the tests were done by selecting the element on position 1000000) were constantly equal to 0.

As a conclusion, these experiments can show that low theoretical asymptotic complexity can prove to be very cost-inefficient due to constants and that this algorithm can be used for small search spaces that are repeated often or if the use case requires repeated insertions and lookups in a sorted array.

Github link to code: https://github.com/Viktor-Teodor/TrieForSorting/tree/main