Algorithm Analysis — Part 1: Learning to analyze algorithms
By: Mateus Melo
Tags: Computer Science, Programming, Algorithm Analysis, Algorithms

We use algorithms and data structures in order to create efficient solutions for problems that we face. It’s obvious that there are better ways to do things. So, when facing a problem, there are algorithms that are better than others. But how exactly do we tell that one algorithm is better, or even if it is the best? And even if we have this answer, can we generalize it? Can we create a method that we’ll be able to use in order to compare all algorithms out there?

We start with the idea that there are algorithms that are better than others for certain situations. This is simple and intuitive. When searching for your keys to go to work, you don’t look in places you haven’t been. It’s more efficient to visit places you know you visited. But how does this translate to the context of machines and algorithms? How can we tell that an algorithm is better than another one?

Each situation has its own unique characteristics and requirements. An algorithm that is good by a set of standards might not be as good when analyzed through another one. Still, our goal here is to develop a method that will work with most algorithms.

What we can do is abstract the many different factors that might go into considering an algorithm good or not and stick to a few that seem relevant enough to any algorithm. Those factors are time and memory. This makes a lot of sense, since solutions that are faster or that consume more memory translate to less cost. From here on out, we’ll focus on analyzing the running time.

This was our first step. We have parameters that can help us in comparing two algorithms. But this isn’t enough. We don’t know how to determine these parameters. What factors determine the running time of an algorithm? How can we calculate it? When saying that one car is faster than another, we cap specify a multitude of things that determine that fact. Maybe the engine is more modern, maybe its design is more aerodynamic, or it might have a larger cylinder capacity. These are concrete reasons that make a car faster. And with that, we gain deeper insight on the problem. But what exactly are those reasons for algorithms?

In the real world, there are, again, many factors that may come into play. In a system, the hardware, software, operating system and many more could give their own contribution to the running time of an algorithm (or memory consumption). Just like when we defined our parameters, we need to abstract this multitude of things and focus on what really matters, which is the size of the input data. This also makes a lot of sense, since a problem’s size and complexity grows as its input grows. If we’re designing an algorithm that performs a search on an array, it will take a lot more to time to find a result within an array with 10 million elements than it would within another one with only 10.

We have taken our second step. So far, our model analyzes algorithms by their running time, which depends on the size of the input. We can say that the running time T is a function of n. Therefore, T(n) is an algorithm’s running time for an input of size n.

Great. But given a particular algorithm, how can we calculate its running time?

An immediate response could be that we can just run the algorithm for a number of different input sizes while timing it. Then, we can plot the pairs of values and analyze how the algorithm grows. This would be a valid method. Nevertheless, this can get really tiresome and it’s not broad enough. When running this experiment, we are still very dependent on the specifications of the system in charge. We fall back into the same problem before our second step.

What we need is a mathematical method that will enable us to analyze algorithms without having to relentlessly test them. Is such a model possible?

Return to the roots

In the 20th century, computer scientists were not sure if such accurate models were possible. Then, legendary computer scientist Donald Knuth — a.k.a. The Father of Algorithm Analysis — stated that they are indeed possible. By describing algorithms as a series of instructions and operations, we can assign each of these an individual contribution to the running time (cost), and then count the amount of time they occur (frequency). By multiplying both, we get the cost of a certain operation. By summing all costs, we can calculate an algorithm’s running time. It’s important realize that we are not totally free from the specifications of the system, as a particular operations have their cost defined by it. But what we can do is represent the value with an arbitrary constant (Cn), since we don’t want to know its exact value. Let’s see an example.

In this very simple piece of C++ code, we’re looping through an array and counting how many of its elements are equal to zero. As simple as it may be, we have a few different operations that are happening. There is assignment, inequality comparisons, equality comparisons, increment and array access. When we count them, we get the following frequencies:

Now, let’s calculate the running time.

We can now see that the algorithm’s running time grows linearly according to its input.