In the last part we discussed evolution of the interrupt delivery process from the devices in the x86 system (PIC → APIC → MSI), general theory, and all the necessary terminology. In this practical part we will look at how to roll back to the use of obsolete methods of interrupt delivery in Linux, and in particular we will look at Linux kernel boot options: Also we will look at the order in which the OS looks for interrupt routing tables (ACPI/MPtable/$PIR) and what the impact is from the following boot options: You've probably used some combination of these options when one of the devices in your system hasn't worked correctly because of an interrupt problem. We'll go through these options and find out what they do and how they change the kernel '/proc/interrupts' interface output. In this article for our interrupt investigation we will be using custom board with the Intel Haswell i7 CPU with the LynxPoint-LP chipset which runs coreboot. We will be getting information about interrupts in the Linux system through the command: Here is the output when the kernel was booted without any external options: File '/proc/interrupts' is the procfs Linux interface to the interrupt subsystem, and it presents a table about the number of interrupts on every CPU core in the system in the following form: Everything here is like it is supposed to be in the modern system. For the devices and drivers which support MSI/MSI-X, this is the type of interrupt that they use. The rest of the interrupt routing is done through the APIC controller. Simplistically, the interrupt routing schematics can be drawn like this: (red lines are active routing paths and black lines are unused routing paths) A device that supports MSI/MSI-X interrupts should have that particular capability listed in its PCI configuration space. As an example of that let's look at a little fragment of the lspci output for the devices that declare they use MSI/MSI-X. In our case it is a SATA controller (interrupt 'ahci'), two ethernet controllers (interrupts 'eth58*' and 'eth59*'), graphical controller ('i915'), and two HD Audio controllers ('snd_hda_intel'). As we see, all of these devices either have a string «MSI: Enable+» or «MSI-X: Enable+». Let's downgrade our system! For a start let's boot with the kernel option 'pci=nomsi'. Because of this option MSI interrupts become IO-APIC/XT-PIC depending on the interrupt controller in use. In this case the priority choice is still modern APIC controller, so the interrupt picture will be: Output of /proc/interrupts: As expected, all MSI/MSI-X interrupts have disappeared. Instead of them devices now use interrupts of 'IO-APIC-fasteoi' type. Let us draw our attention to the fact that earlier, before enabling this kernel boot option, each of the 'eth58' and 'eth59' had nine interrupts! But now each of them has only one interrupt. Recall that without the MSI, one function in the PCI device can have only one interrupt! Here is a little info from the 'dmesg' command about the ethernet controllers' initialization:  — boot without the 'pci=nomsi' option:  — boot with the 'pci=nomsi' option: Because of the decreased number of interrupts per device, enabling this option can lead to a significant performance limitation of the device driver, and that is not even counting that according to the Intel research 'Reducing Interrupt Latency Through the Use of Message Signaled Interrupts', MSI interrupts 3 times faster than the IO-APIC interrupts and 5 times faster than the PIC interrupts. This option disables I/O APIC. MSI interrupts can still find their way to all of the CPUs, but the rest of interrupts from the devices can go only to CPU0, because PIC is only connected to CPU0. However, LAPIC is working and all other CPUs can still work and handle interrupts. As we see, all IO-APIC-* interrupts have turned into XT-PIC-XT-PIC, and all of these interrupts have been routed to CPU0 only. MSI interrupts on the other hand have remained unchanged and go to all of the CPUs. This kernel boot option disables LAPIC. MSI interrupts can't work without LAPIC, and I/O APIC can't work without LAPIC either. All of the device interrupts can only go to the PIC, and it works with the CPU0 only. And without LAPIC the rest of the CPUs besides CPU0 won't work. Output of /proc/interrupts: Actually there is only one combination for the new variant of routing: «noapic pci=nomsi». In this case all interrupts from the devices only go to the CPU0 through the PIC controller. But the LAPIC system is still working, so all the other CPUs can work and handle interrupts. You cannot combine any other options with «nolapic» since it makes I/O APIC and MSI unaccessible. Therefore, if you've ever added Linux kernel boot options like «noapic nolapic» (or the most common case «acpi=off noapic nolapic») it seems like you've written some extra letters. Finally, here is the result of the options «noapic pci=nomsi» to our interrupt routing picture: And the output of /proc/interrupts is: How does the operating system get information about the device interrupt routing? The BIOS prepares such info for the OS in the form of: It is worth to note for the MSI interrupts declaration that the BIOS doesn't need to do anything extra (beside declaring the use of the LAPIC): all the aforementioned routing information is needed only for the APIC/PIC interrupt lines. Tables in the list above are presented in the order of priority. Let's examine it in detail. Let's assume the BIOS has presented all this data and we boot our OS without any extra boot options: In the case when we boot with the option noapic: If ACPI tables aren't present or interrupt routing with ACPI is disabled through the option acpi=noirq or pci=noacpi (or ACPI subsystem is completely disabled with the acpi=off option), then the OS looks for the MPtable (_MP_) to get all the interrupt routing information: If ACPI tables aren't present or interrupt routing with ACPI is disabled through the option acpi=noirq or pci=noacpi (or ACPI subsystem is completely disabled with the acpi=off option), and if the MPtable (_MP_) is not present either (or there is a boot option noapic or nolapic): If there is no $PIR table or it is not full, then the OS will look at the registers 0x3C/0x3D of the device's PCI configuration space to guess interrupt routing. Here is a picture summarizing all of this: One should remember that not every BIOS provides all of these three tables (ACPI/MPtable/$PIR), so if you've passed an option to your bootloader (e.g. GRUB) that disables the use of ACPI or ACPI and MPtable for the interrupt routing, it is possible that your system won't boot. Note 1: In the case when we try to boot in APIC mode with the option 'acpi=noirq' and without MPtable present, the picture of interrupts will be like in the case of normal booting with only the 'noapic' option. The operating system will go to PIC mode by itself. In the case when you try to boot without any ACPI tables at all ('acpi=off') and without MPtable present, then the picture will be like this: This happens because without the ACPI MADT table (Multiple APIC Description Table) and the necessary info from the MPtable, the operating system doesn't know APIC identifiers (APIC IDs) for the other CPUs and can't work with them. But the LAPIC of the main CPU0 works because we haven't disabled it, and MSI interrupts can still go to it. So the interrupt picture would be: Note 2: In general, interrupt routing with the use of ACPI in an APIC case should match the interrupt routing with the MPtable. Also, the interrupt routing with the use of ACPI in a PIC case should match the interrupt routing with the $PIR table. Therefore the '/proc/interrupts' output should not differ. But in my investigation I've noticed one strange fact. For some reason in the case of interrupt routing through the MPtable there is a cascade interrupt «XT-PIC-XT-PIC cascade» in the output: It is a little bit strange that it happens like that, but it seems like  the kernel source documentation says that it is OK. In conclusion we review for one more time the discussed options. Interrupt controller choice options: Interrupt tables priority options: In the next part we will look at how coreboot configures the chipset for the interrupt routing. Special thanks to Jacob Garber from the coreboot community for helping me with this article translation Инженер-программист