# Hardware Locality: Peering under the hood of your server

Brice Goglin, Jeffrey M. Squyres, Samuel Thibault

► To cite this version:

Brice Goglin, Jeffrey M. Squyres, Samuel Thibault. Hardware Locality: Peering under the hood of your server. Linux Pro Magazine, 2011, 128, pp.28-33. inria-00597961

HAL Id: inria-00597961

https://inria.hal.science/inria-00597961v1

Submitted on 31 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# hwloc (Hardware Locality): peering under the hood of your server

Brice Goglin, Jeff Squyres, and Samuel Thibault

linuxpromagazine.com issue 128, July 2011

Internal server topology is getting more and more complex; understanding where your processes are running in your server can have a major impact on overall system performance.

You just replaced your creaky, old, periodically-reboots-for-no-reason server with a shiny, modern server. The new hotness has faster disks (and more of them), 4x the amount of RAM, and is generally all around mo'bettah. Woo hoo!

But are you getting all the performance that you should? Exactly where is all that memory that you got, anyway? Do you know how big your shiny new L1, L2, and L3 caches are, and how they are shared across processor cores and sockets?

## 1 location, Location, LOCATION!

Most modern commodity servers have a non-uniform memory architecture (NUMA), meaning that the RAM is distributed between the processors on the machine. The processors are, in turn, connected by a high-speed network (e.g., QPI on Intel-based servers and Hypertransport on AMD-based servers).

The open source Hardware Locality project<fnref n="1" />, a sub-project of the larger Open MPI community<fnref n="2" />, is a set of command line tools and API functions that allow system administrators and C programmers to examine both the NUMA topology and provide details about each processor and memory type in a server.

Plus, it draws pretty pictures! Managers love pretty pictures (see Figure 1).

## 2 Background: NUMA

When a process is running on a specific processor, reading and writing to memory that is directly attached to that processor is always fastest. Reading and writing to memory attached to a remote processor is both slower and consumes

<footnote n="1">¹http://www.open-mpi.org/projects/hwloc/</footnote>

<footnote n="2">²http://www.open-mpi.org/</footnote>
<figure><img src="image_2.png" /><figcaption>Figure 1: Sample output from hwloc showing a two socket / four core server with 16GB total RAM. "PU" stands for "processing unit", or hardware thread ("hyperthreads" in Intel processors).</figcaption></figure>

bandwidth across the inter-processor network. Specifically: reading and writing to remote memory can even detract performance from *other* processes on the server.

Linux does a reasonably good job of trying to keep related processes and their memory local to each other. Usually. System administrators and C programmers can help Linux by giving specific directives about where they want processes and their associated memory to be located.

Proper process and memory placement can have a dramatic effect on both the performance of individual processes and the overall system.

## 3 (Non-)Portability of Server Topologies

Before you can place processes and their memory, you first need to know what the internal topology of your server is. Your server's topology is actually a network of various entities, which hwloc calls *objects*.

Figuring out the topology of that network is hard, as the object numbering varies depends on factors such as the motherboard vendor, the BIOS version, and even with the operating system version. Changing any of these factors may result in the objects being renumbered. Ouch!

For example, Figures 2 and 3 show the same server but with the processor core objects ordered differently.

Most operating systems – including Linux – offer methods of querying processor ordering and details of the server's internal topology. Unfortunately,
<figure><img src="image_3.png" /><figcaption>Figure 2: A quad-core processor whose cores and linearly numbered.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Figure 3: A quad-core processor whose core numbers are interleaved.</figcaption></figure>
```
$ cat /proc/cpuinfo
[...]
processor : 3
[...]
physical id : 0
siblings : 4
core id : 3
cpu cores : 4
$ cd /sys/devices/system/cpu/cpu3/topology
$ cat core_id
3
$ cat core_siblings
00000000,0000000f
```

<caption>Listing 1: Linux topology information on x86 architectures.</caption>

these methods are notoriously non-portable.

For example, the information returned from Linux's `/proc/cpuinfo` and `/sys` virtual filesystem is not necessarily easily human-readable, and may be slightly different between various platforms and versions of the Linux kernel. See Listing 1 for one example of how Linux outputs this information.

## 4 Overview

The Hardware Locality tool suite takes care of hiding all these technical and non-portable details behind a convenient interface. The hwloc core gathers all the available pieces of information from the operating system and builds an abstracted, portable tree of objects: memory nodes (and node groups), caches, processor sockets, processor cores, hardware threads, etc.

This tree can be obtained via hwloc's CLI or C API. Figure 1 shows the graphical output from hwloc's `lstopo` command on a machine with two NUMA nodes, each with a processor socket containing two cores. Each core has two hardware threads (known as "Processing Units" in hwloc terminology).

hwloc's tools (such as `lstopo`) not only provide the object numbering as seen by the OS (so-called "physical numbers"), but also compute a stable, logical numbering. This "logical" ordering runs linearly through adjacent objects and can thus be used to reliably describe which object is targeted, even if a server's BIOS or operating system is upgraded.

hwloc reveals its true power when one starts using its C programming interface. The entire tree of objects is presented in a cross-linked data structure; the tree can be traversed via children, parents, siblings, and cousins.

Once an application finds the object(s) that it is looking for in the tree, hwloc's tools can bind the execution and/or memory allocation of processes or
threads to any (combination of) these objects, using command-line tools or the C API.

# 5 Installation

hwloc is starting to get picked up by various Linux distributions (e.g., Debian Squeeze, Ubuntu 10.04, FC14, RHEL 6.1), but it isn't available everywhere yet. As such, you may need to download and install hwloc manually. The most recent stable release can be found at:

http://www.open-mpi.org/software/hwloc/current/

Once downloaded, hwloc installs with the common "configure / make all install" procedure:

```
$ ./configure [--prefix=/there]
[...]
$ make all
[...]
$ sudo make install
[...]
```

You can optionally specify a non-default installation location with the `--prefix` option. hwloc is not dependent on any particular installation path; you can even safely install multiple copies/versions of hwloc into distinct prefixes on a single system.

Although a lot of effort has been put into making the `configure / make` steps "just work", sometimes you need to specify some additional parameters to `configure` to get what you need.

Running `configure --help` shows a full list of the command line switches that `configure` accepts, such as options to enable / disable XML, graphics support, etc.

# 6 Command-Line Tools

hwloc includes several command-line utilities. `lstopo`, described above, is perhaps the most commonly-used. It can output in plain text, prettyprint text, colored ANSI text, X11, FIG, Postscript, PDF, PNG, SVG, and XML formats. Listing 2, for example, shows the plain text output from `lstopo`.

Only the resources that are actually available to the current process are displayed by default. Figure 4 shows a sample server with one core offline. Other resources (e.g., stopped or restricted cores) can also be displayed by giving the `--whole-system` option.

`lstopo`'s XML export dumps the entire topology so that it can be used by other scripting tools (e.g., Perl scripts), or even loaded in `lstopo` on another server. The example below first exports the machine topology as seen on `host2`, transfers the resulting XML file back to `host1`, and then creates a PDF file.
```
Machine (24GB)
NUMANode P#0 (12GB)
Socket P#0
  Core L#0 + PU L#0 (P#0)
  Core L#1 + PU L#1 (P#2)
  Core L#2 + PU L#2 (P#4)
  Core L#3 + PU L#3 (P#6)
NUMANode P#1 (12GB)
Socket P#1
  Core L#4 + PU L#4 (P#1)
  Core L#5 + PU L#5 (P#3)
  Core L#6 + PU L#6 (P#5)
  Core L#7 + PU L#7 (P#7)
```

<figure><img src="image_5.png" /><figcaption>Figure 4: `lstopo` output where core `P#0` is not available to the current user (e.g., restricted via Linux `cpuset/cgroup`), and core `P#1` is offline.</figcaption></figure>
```
host1$ ssh host2
host2$ lstopo h2.xml
host2$ exit
host1$ scp host2:h2.xml .
host1$ lstopo -i h2.xml h2.pdf
```

In addition to writing to files, if the `DISPLAY` environment variable is set, `lstopo` draws the topology in an X11 window.

`lstopo` can also filter the displayed topology to ignore some objects, display more verbose textual outputs (including all object attributes such as cache sizes), etc.

After consulting the topology tree, `hwloc` can then be used to bind processes or threads to specific cores or sockets.

The `hwloc-bind` utility offers a command-line binding interface that benefits from `hwloc`'s advanced knowledge of the topology. Contrary to usual tools such as `taskset` or `numactl` that use physical processor and memory indexes, `hwloc-bind` lets you bind tasks using logical (or physical) object identifiers.

In the examples below, `prog1` will run on any processing units in the second logical socket. `prog2` will run on any processing units in the third or fourth logical core, while its memory will be allocated on the first logical NUMA memory node.

```
$ hwloc-bind socket:1 -- prog1
$ hwloc-bind core:2-3 --membind numanode:0 -- prog2
```

Binding individual processes to specific locations is useful, but using binding as a tool to load balance an entire server is more useful.

`hwloc`'s `hwloc-distribute` was designed to compute where to bind processes depending on how many you want to distribute across the server. It returns bitmasks (i.e., a list of physical processing units) that can be given to `hwloc-bind`. Listing 3 is an example showing how four programs can be easily bound to the four cores of the whole machine in Figure 1 by using `hwloc-distribute`'s output.

Note that Listing 3 shows that each process will be bound to *two* Linux virtual processors. These processors correspond to the two hardware threads (i.e., processing units) in each core in Figure 1.

`hwloc` also lets you check if your processes are properly bound, as shown in Listing 4. `hwloc-bind` can return the current binding of any process. The `hwloc-ps` utility lists all processes that are currently bound to part of the machine. `lstopo --ps` inserts those processes within the topology output.

Finally, `hwloc` also includes a Swiss army knife called `hwloc-calc` which serves as a generic conversion tool between miscellaneous `hwloc` objects. As shown in Listing 5, it accepts objects or bitmasks as input, combines them, and outputs the result as bitmasks, objects, list of indexes, etc. Its outputs can be given to other `hwloc` utilities or used in scripts.
$ hwloc-distribute 4
0x00000011
0x00000022
0x00000044
0x00000088
$ hwloc-bind 0x00000011 prog1
$ hwloc-bind 0x00000022 prog2
$ hwloc-bind 0x00000044 prog3
$ hwloc-bind 0x00000088 prog4

<caption>Listing 3: Easy process distribution via hwloc-distribute.</caption>

$ hwloc-bind --get 7714
Core:2
$ hwloc-ps
7712 PU:1 prog1
7713 PU:3 prog2
7714 Core:1 threadedprog
$ lstopo --ps
# ... topology output ...
Core P#2
7714 threadedprog
PU P#1
7712 prog1
PU P#3
7713 prog2

<caption>Listing 4: Getting a process' binding.</caption>
# Output a mask containing
# cores 0 and 1
$ hwloc-calc Core:0-1
0x00000033

# Largest objects containing
# these cores
$ hwloc-calc Core:0-1 --largest
Socket:0

# List of physical indexes of
# hardware threads (Processing
# Units) contained in these cores
$ hwloc-calc Core:0-1 --intersect PU --physical-output
0,4,1,5

# List of physical indexes of
# second hardware threads of all
# cores
$ hwloc-calc Core:all.PU:1 --intersect PU --physical-output
4,5,6,7

Listing 5: hwloc-calc swiss-army knife in action.
```c
hwloc_obj_t obj;
depth = hwloctopology_get_depth(t);
for (d = 0; d < depth; d++) {
    n = hwloc_get_nbobs_by_depth(t, d);
    for (i = 0; i < n; i++) {
        obj = hwloc_get_obj_by_depth(t, d, i);
        printf("%d,%d type %d\n", d, i, obj->type);
    }
}
```

Listing 6: Printing the whole topology, row by row, in an array-of-array style.

```c
n = hwloc_get_nbobs_by_type(topo, HWLOC_OBJECT_CORE);
obj = hwloc_get_obj_by_type(topo, HWLOC_OBJECT_CORE, n - 1);
hwloc_set_cpubind(topo, obj->cpuset, 0);
```

Listing 7: Directly accessing a given object type (e.g., binding this process to last processor core).

# 7 C Programmation Interface

A set of core functions is provided in `hwloc.h`, along with a large set of inline helpers in `hwloc/helper.h`.

It its simplest form, the initialization and detection of a topology can be reduced to two function calls to fill a `hwlocTopology_t` variable:

```c
hwlocTopology_t t;
hwlocTopology_init(&t);
hwlocTopology_load(t);
```

`t` is the head of a tree of objects which can be traversed many ways, including array-of-array style (see Listing 6) or direct access a specific type of object (see Listing 7).

In Listings 6 and 7, the `type` and `cpuset` members are just two examples of the variety of information available at each object in the tree.

Listing 7 also shows the use of `hwloc_set_cpubind()` to bind the current process to the specified `cpuset`. `hwloc`'s full documentation (in PDF, HTML<fnref n="3" />, and man page form) describes many more functions and options, such as those to bind threads, bind other processes, migrate memory, etc.

Finally, it should be noted that the `hwloc` community has created Perl and Python bindings to make this internal programming interface available through other languages. Awesome.

<footnote n="3">³http://www.open-mpi.org/projects/hwloc/doc/</footnote>
# 8 Embedding hwloc Within Your Software

In most cases, applications utilizing the hwloc C API can simply link against a system-installed hwloc. However, this may not always be the case.

Consider an example software package named Whazzle. Whazzle both requires hwloc and has been qualified with a specific version of hwloc. For these two reasons, the Whazzle developers decided to "embed" an entire copy of an expanded hwloc tarball in the Whazzle source code tree.

The presence of hwloc in the Whazzle code base ensures that hwloc will always be available on the target system – regardless of whether hwloc is installed on the system or not. It also guarantees that Whazzle uses the one, specific version of hwloc which has been developed and tested.

hwloc's embedding design works best with software projects that use GNU Autoconf, Automake, and Libtool. Whazzle fortunately uses these tools, so the Whazzle developers then use the hwloc-provided M4 GNU Autoconf macros in their configure.ac script.

As such, when Whazzle's configure script is invoked, hwloc will also be configured. And when make is invoked with any of the standard Automake targets (e.g., "all", "clean", "install", etc.), they also traverse into the hwloc directory and perform the appropriate actions.

# 9 Upcoming Features

The recently released version 1.2 of Hardware Locality [4] focused on exposing physical distances between objects (especially NUMA nodes) to applications that need such information to optimize their placement. The ability to restrict the hwloc topology to a custom part of the machine was also added. It is now also possible to find out where a process or thread actually ran, as opposed to where it is bound (e.g., if a single-threaded process is bound to an entire socket; it likely only ran on a single processing unit).

The next major release series of hwloc (1.3) will focus on adding I/O devices to the topology because the performance of applications not only depends on where tasks are running and memory buffer are allocated, but also on which I/O peripherals they access (see Figure 5).

Additionally, v1.3 will enable process/memory placement according to I/O locality, as shown in Listing 8.

# 10 The Final Word

The real power of the hwloc tool suite is that it provides a portable mechanism to examine and utilize server hardware topology. Developers can effect topology-based decisions in their applications; system administrators can load balance their servers.

Indeed, the hwloc community is growing. A variety of software projects have included hwloc support, including (but not limited to): Open MPI, MPICH2,
<figure><img src="image_6.png" /><figcaption>Figure 5: A two socket / eight core server with PCI devices.</figcaption></figure>
# Starts io_program on a core
# near the sda disk
$ hwloc-bind os=sda io_program

# Starts web_server on a core
# near the eth2 interface
$ hwloc-bind os=eth2 web_server

# Starts myProg near the PCI
# device whose bus ID is 02:04.0
$ hwloc-bind PCI02=04.0 my Prog

Listing 8: Binding processes near I/O devices (in the upcoming hwloc 1.3 release series).

MVAPICH2, Padb, ForestGOMP, StarPU, PLASMA, DAGuE, Qthreads, the Rose compiler, and the Open Grid Scheduler.

Remember: as core counts in servers continue going up, it's all about location, Location, LOCATION!

## 11 The Authors

**Dr. Brice Goglin** has been a Computer science researcher at INRIA since 2006. He has been working on several open-source software projects for ten years, including high-speed networking stacks for message passing in parallel applications, and also on Linux kernel support for high-performance computing.

**Dr. Jeffrey M. Squyres** has been a software engineer at Cisco Systems, Inc. since 2005. Jeff is Cisco's core software developer in the open source Open MPI project, of which Hardware Locality (hwloc) is a sub-project. He has worked in the open source High Performance Computing (HPC) field since his early graduate-student days in the mid-1990's.

**Dr. Samuel Thibault** has been a Computer science researcher at University of Bordeaux since 2008. He has been working on various open source projects for ten years, and notably on thread scheduling since 2004, in particular on the Marcel user-level thread library which happened to be the nest for the ground ideas and portability code of hwloc. He is currently focusing on scheduling GPU computation and transfers.