The Memory Hierarchy
By: Joseph Mellor
Tags: Optimization, Coding, Optimization Techniques, Programming

Explaining the Allegory

Designing how your computer manages memory is almost exactly like figuring out an efficient way to have everything you need for a class. I’m going to list all the analogous concepts between the allegory and the computer, and then I’m going to explain them immediately. If you’re confused by the computer terms, think entirely in terms of the allegory, then just replace the allegory terms with the computer terms one step at a time.

The professor is the CPU .

is the . Her hands are the registers .

are the . Her classes are her programs .

are her . Her labels and labeling system are memory addresses .

are . Her office is RAM .

is . Her backpack is the L2 cache .

is the . The magnets and marker basket are the L1 cache .

are the . Her house is secondary storage.

CPU

The Central Processing Unit (CPU for short) is the brain of a computer. It decides what to do next, how to organize things, etc. I could say a lot about it, but treat it just like the professor.

Registers

Let’s say you want to make a computer. First, to do anything on the computer, you have to have some data stored inside the CPU itself, otherwise the CPU can’t interact with data at all. Just as the professor needed some way to interact with what she brought to class, so your computer needs some way to interact with its data. You decide that you’re going to give the CPU some registers, each of which consists of a few bytes of memory. Accessing memory from registers is almost instantaneous, but you can’t have too many registers for reasons involving the specifics of hardware. In general, you will have anywhere from four to sixty four (or some other power of two) registers.

Programs

The actual content of the classes determines what stuff you’ll need to bring to each class, just as a program determines what the CPU needs to do.

Memory Addresses

You’ll also need a way to identify which data you want, so you need to come up with a labeling scheme like the professor. The labeling scheme should consist of a list of what you want to find and where it is. To keep track of where it is, you use a memory address. A memory address is a number that corresponds to a specific location in memory. On a computer, each memory address is a specific number that uniquely identifies a byte of memory.

RAM

Now, you can’t store an entire program in the registers alone in the same way the professor couldn’t just hold everything she owned in her hands, so you need to make a place to store information such as files and the values of variables. You decide to make a component of the computer that can store a lot of memory, which you call Random Access Memory or RAM for short. Since computers have to be way more organized than an office, you’re going to make RAM one continuous list of bytes and use a memory address to access specific bytes.

The earliest modern computers just had registers and RAM for memory. In fact, a lot of computers like calculators (at least up to the TI-89 Titanium) and early video games just had RAM, which meant that you would lose your data if you turned them off. Old games like Metroid used to have passwords that would indicate your progress specifically because they couldn’t save the data (The Legend of Zelda was an early exception because it had save files.).

Random Access?

In STEM, random means you cannot tell me with certainty what’s going to happen next. For example, if I flip a coin, you cannot tell me if it’s going to be heads or tails, even if you could tell me the likelihood of each outcome. In another example, if I were to roll two dice, I can tell you that the sum will be seven around one out of every six rolls, but I can’t tell you what the next roll will be.

The term Random Access X just means that the computer does not know what element of X will be accessed next. Random Access Memory means that the computer doesn’t know what memory address you’re going to read from or write to next.

The Caches

The biggest problem with RAM is that it’s slow. Just like going to the office to get something took a while for the professor, getting data from RAM takes a while (for computers, anyway). If you think back to the allegory, her original problem was that she had nowhere to store things she had already taken from the office, so she decided that she would create a place to store it that would have faster access at the expense of not being able to store everything.

In computer science, a cache consists of a place where you store things you’ve already fetched from a larger and slower source of data.

In terms of real world experience, say you’re working on a project or a paper or something else that requires you to do research. Now, you will end up making a bunch of searches that don’t get you anything useful, but say you find a source that you really like. You know that you’re probably going to refer back to it later, so you’ll want to somehow save a reference to it, whether it includes bookmarking the page, copying the link, or whatever. In saving something you have already fetched from a larger (the internet has way more memory than your computer) and slower (compare all the time you spent searching for the source to the time it takes to find it in a list of bookmarks) source of data, you have cached the source.

For a CS example, let’s say you’re running Twitter and you notice that a lot of people are searching for a specific hashtag, #memoryhierarchy for instance. Instead of having each user searching for #memoryhierarchy send a request to go through all your servers searching for tweets with #memoryhierarchy, have one user do it, store the results of that search on a bunch of servers, then whenever someone searches for #memoryhierarchy, just send them the results you’ve already found without going back through all the servers. Don’t waste time finding something that’s already been found or doing something that’s already been done.

In the example, the backpack and the magnets were two different levels of cache. Getting stuff you need from the backpack takes less time than getting stuff from the office, but the backpack can’t hold nearly as much as the office. Likewise, getting stuff you need from the magnets takes less time than getting stuff from the backpack, but the magnets can’t hold nearly as much as the backpack.

Accessing memory from the L1 Cache is faster than accessing memory from the L2 Cache, but you can’t store as much in the L1 Cache. Likewise, accessing memory from the L2 Cache is faster than accessing memory from RAM, but you can’t store as much in the L2 Cache. On modern computers, you often have an L3 Cache between the L2 Cache and RAM, which follows the same pattern of having more memory than the L2 Cache but with a slower access time.

People have already written articles about caches and performance that contain more specific information, so I’ll just link an article for you to read. I would focus on the relative speed of the different levels of memory in that article in particular.

Secondary Storage

What’s the only type of memory I haven’t mentioned yet? That’s right, your hard drive or solid state drive. RAM consists of data that your program has requested and your secondary storage consists of data that your program might request in the future, such as essays you’ve written, textbooks that you downloaded legally, pictures, etc. If it’s a file on your computer, it’s in secondary storage. Technically, we can extend the idea of secondary storage to anything that isn’t RAM, caches, or registers such as a server, but you get the idea. Anyway, secondary storage corresponds to the professor’s house or anywhere else she might store stuff that she could bring to her office but won’t for sake of space.

Why Have a Memory Hierarchy?

I hope at this point you see the general pattern: more memory means less speed. Of course, the memory-speed trade-off isn’t a fundamental law of nature, but then your computer would be way more expensive since faster memory is more expensive (time is money, after all). The memory hierarchy is a nice compromise between cost, memory, and speed. If you write good programs that use the memory hierarchy to your advantage, you can effectively simulate having a lot of high speed memory.

How to Use the Cache

As a programmer, you do not move things back and forth between caches and RAM, your computer handles that for you. The computer, however, does it in a reliable way that you can use to your advantage. In the following subsections, we’re going to discuss the three ways you can optimize your cache use.

These next two principles are known as the Principles of Locality.

Temporal Locality

Let’s go back to the allegory before the professor bought her backpack. She spent most of her time going back and forth between her office because she kept switching between things she needed. She used the syllabus first, then her markers, then the syllabus, then the textbook, then the markers, then she kept switching back and forth between the markers and the textbook until her class ended. We can come up with a way she could have sped up her class by adding another rule: once she takes something from her office, she cannot use anything else until she has completely finished using what she currently has. In her case, she could have discussed the syllabus in its entirety without stopping to draw anything on the board, then gone back to her office to trade the syllabus with the textbook, then discussed everything relevant from the textbook, then gone back to her office to get the markers, then used the markers. If she had, she would have only gone back to her office twice, meaning she would have only wasted ten minutes.

In the same way, using the cache to your advantage is quite simple: once you access some data, do everything you can with the data before getting new data.

Now, you might come up with the reasonable objection that she probably would have needed to switch more often than just when she finished using something. I have assumed that she could do what she needed to do with her markers without having to refer back to the syllabus or the textbook and vice versa. The general principle of doing everything you can with what you have still applies, but it just turns out that she couldn’t do everything she needed to do with a resource until she finished. If you as a programmer just had registers and RAM, you would also probably need to switch between resources. Since we have more than just RAM and registers, let’s add the cache into the mix.

Whenever you access data, the computer will copy it into the fastest memory it can because it thinks you’re probably going to use it again in the future, which is known as Temporal Locality. For example, if you’re calculating an average value, you’ll want the memory for the running total to be accessed quickly, so you’ll probably store it in either a register or the L1 cache. Once you’re done calculating the average, you might not need the total anymore, so your computer is going to leave it in the L1 cache until the CPU needs the space for something more important. It will then demote the data for the total down to the L2 cache. If the running total gets used again, it will move back into the L1 cache, otherwise, it will stay in the L2 cache until the program needs the space for something more important. It will then move the data for the total down to the L3 cache and so on until it reaches RAM, where it will either be used again and moved back up to the L1 cache or the program will tell the computer it’s done using the memory.

In other words, if you use data once, it will be in the high speed memory until something else replaces it, meaning you will pay no speed penalty if you use the data soon after.

As a real albeit simplified example, let’s say we have an AI that will find text and faces in a set of small images. Furthermore, let’s say we have a hundred images to check and we can only fit one 4 KB image in the L1 cache at a time. If we were to write the following code:

for image in images:

find_text(image)



for image in images:

find_faces(image)

we would end up loading each image once for each for loop, meaning we load an image into the cache 200 times. On the other hand, if we were to write this code:

for image in images:

find_text(image)

find_faces(image)

when we run find_faces , we still have the image in the cache from when we accessed the image in find_text , meaning we would only load an image into the cache 100 times. Most of the time, using temporal locality to your advantage is trivial, and so I would not consider it a premature optimization.

Spatial Locality

When the professor first started with her labeling scheme, she originally didn’t organize her office at all. For example, to find everything she needed for her third class, she would have to check in the leftmost drawer under her desk for her markers, take the third book from the left on the top shelf of her bookshelf, take her computer off her desk, and then get the documents from her class from the fifth binder from the top of her pile of binders. Her other classes had similar requirements. Since she was quite descriptive, it wasn’t hard for her to find everything that she needed, but it still took a while to find out where to go in her office, getting what she needed, and put it in her backpack for each resource. Since she wanted to speed up the process of going to class, she decided that she would group together all the resources she needed by class, not by type. She decided to put her documents, the textbook, and anything else she needed for each class in the same box so that she could pick up the box, move it next to her backpack, then take what she needed from the box and put it into her backpack.

Of course, this system wasn’t perfect. Sometimes, different classes needed different resources but she couldn’t put them in both boxes at the same time, so she decided to put the resource into the box for the class that used the resource most often. For example, her third class used her laptop way more often than her other two classes, so she put it in the box for her third class. Other times, she had to put some resources for different classes into the same box to save space. All things considered though, the boxes made her more efficient.

The computer uses the same exact idea to make sure the memory you want to use is in the cache: if you load in some memory, it will try to load in the neighboring memory too in what is known as a cache line, which means that if you want to use the cache to your advantage, keep related data together.

In my description of the professor and her boxes, I modified slightly what the computer does to make the allegory make sense. Your computer will not look through the boxes and figure out what you need to put in the cache. Instead, it would put the entire box into the backpack regardless of whether the professor needed everything in the box. Knowing this fact, you should make sure that you fill your boxes with exactly what you need.

As a real world example, let’s say that you’re programming 2D video game and you want to find out if two objects are colliding. For example, if a character is colliding with a wall, he, she, or it should stop moving. If a character collides with an enemy projectile, the character should take damage. To tell if two objects are colliding, all you need to know is where the collision area or hitbox of the objects are in relation to each other.

While you could store the hitbox within the object it represents (such as a player or a projectile) alongside other data like health, stamina, inventory, etc., the hitbox is only relevant to collision detection. Instead, you can store all the hitboxes together. Since a 2D hitbox only contains four numbers for the left, right, top, and bottom sides of the box and each number can be stored in a four byte float, you can store sixteen hitboxes in a single 64 byte cache line. If you were to store the hitbox within the collision objects, you would end up having to load way more memory since all the hitboxes are in different cache lines. Using this data layout might, however, be a premature optimization, so you would have to profile your code to make sure you’ll actually see some benefit.

Finite Amount of Memory

For the professor, she should try to fill the box with smaller objects like thinner binders and excerpts from the textbook so she doesn’t have to look through multiple boxes.

Knowing that the computer can only fit so much data in the cache, reducing the amount of memory you use means you can fit more useful memory in the cache, which is the final technique I will discuss in this article. As a practical example, many AAA video games will often use a float (4 bytes) instead of the more precise double (8 bytes) because the extra precision is rarely useful (If you store distance in meters with a float , you can be accurate to the micrometer.) and they don’t want to waste cache memory.

Swap Memory

What happens when the professor can’t keep everything she needs for the day in her office? She has to store some of the stuff in her home, so she decides to store the stuff she won’t use for now back at home and pick it up later when she needs it. Since she doesn’t want her job to be spread out all over the house and waste time finding it, she assigns one room of her house near the door to let her pick up what she needs to pick up quickly.

On you computer, you have swap memory, which functions just like the room in the professor’s house near the door. It’s pretty much auxiliary RAM that’s way slower to use. If you ever require more memory than your RAM has space for, you will end up using swap memory. Your computer is smart about using swap, meaning it will only put things in swap that you’re barely using (such as tabs in your browse that you haven’t looked at in a week) and it will only use swap when it runs out of RAM.

What Happens if I Run Out of Swap?

Don’t.

Seriously, What Happens if I Run Out of Swap?

In general, your computer screen will probably freeze since the computer is spending all its processing power on whatever you’re doing, so it will stop caring about trivial things like changing the pixels of the screen or processing input. Your two options are to wait until it finishes (probably won’t happen) or shut off the computer by holding the power button, as you can’t shut down the computer the normal way.

The answer is still “Don’t” because you would have to intentionally try to run out of swap or deal with such massive amounts of memory that you can expect to run out of swap. For example, the entire human genome needs about 3 GB of memory to store all the base pairs (technically less, but whatever), meaning you could load the entire human genome into memory and still not even run out of RAM. The only practical way to run out of swap is to have way more programs running at the same time on your computer or multiple tabs of Google Chrome.

In terms of the professor, running out of swap means she’s hoarding and needs an intervention.

Other programs

You might be thinking to yourself that your program won’t use swap memory because modern users have at least 8 GB of RAM and you’re only using 5 GB of RAM, but you have to remember that it’s not your program’s memory — it’s the user’s memory. If they want to also run a few other programs while running your program (say to listen to music or look up a tutorial on how to use your program or do multiple things on the internet), they may only be able to offer you 2GB. On the other hand, if you expect users to use your program without using a bunch of different programs at the same time (such as most AAA 3D games), then you’ll be able to use much more memory and users won’t care.

How Did You Get a 4x Speedup?

If you remember the beginning of the article, I said I was able to achieve a massive speedup by first reducing one of the large matrices to just the last two columns then by using two bytes to represent the numbers instead of four bytes, reducing the memory the program used to about a quarter of what it was. In doing so, I was able to fit more numbers into the cache, which made my code faster. Plus, by halving the size of each individual number, the computer could store twice as many numbers in a single cache line, meaning it only had to fetch memory half as often.

I also guarantee that I did everything with the data I could before getting new data throughout the program, but I would have to make this article even longer, so I’ll pass on the example.

Lastly, I looked through the memory usage and made sure that the program wasn’t running slowly just because I was using swap memory (though that is a good reason to use less memory anyway). Initially, I was using around 3GB of RAM and I made sure that as few programs as possible were running.

Summary

Your computer has a memory hierarchy consisting of

Registers Caches RAM Secondary Storage

where registers are the fastest type of memory and secondary storage is the slowest type of memory.

The faster the memory, the less of it you’ll find on your computer.

The memory hierarchy will allow you to simulate having a large amount of fast memory.

The computer will cache memory that you’ve recently accessed and memory that’s nearby.

Writing efficient programs means trying to maximize your use of high speed memory by doing as much as you can with data you’ve already accessed before trying to access more data, keeping related data together, and using as little memory as possible .

. Swap memory exists and you can only run out of it if you try.

Once again, it’s up to you to find out what’s making your program slow and make that part faster. Furthermore, using memory properly will not make poorly written code easy to understand nor will it make a bad algorithm a good algorithm. Using memory properly, however, will make a good program into a great program.