# The Linux Kernel/Multitasking/Real-time
#### RT preemption

The Linux Foundation's Real-Time Linux (RTL) collaborative project is focused on improving the real-time capabilities of Linux and advancing the adoption of real-time Linux in various industries, including aerospace, automotive, robotics, and telecommunications.
Parameter CONFIG_PREEMPT_RT$_{id}$ enables real-time preemption.

#### RT scheduling policies

Scheduling policies for RT:
    SCHED_FIFO$_{id}$, SCHED_RR$_{id}$
       implemented in kernel/sched/rt.c$_{src}$
    SCHED_DEADLINE
       implemented in kernel/sched/deadline.c$_{src}$
API:
    man 1 chrt – manipulate the real-time attributes of a process
    man 2 sched_rr_get_interval – get the SCHED_RR interval for the named process
    man 2 sched_setscheduler, sched_getscheduler – set and get scheduling policy/parameters
    man 2 sched_get_priority_min, sched_get_priority_max – get static priority range

#### Testing RT capabilities

The testing process for Real-Time Linux typically involves several key aspects. First and foremost, it is crucial to verify the accuracy and stability of the system's timekeeping mechanisms. Precise time management is fundamental to real-time applications, and any inaccuracies can lead to timing errors and compromise the system's real-time capabilities.
Another essential aspect of testing is evaluating the system's scheduling algorithms. Real-Time Linux employs advanced scheduling policies to prioritize critical tasks and ensure their timely execution. Testing the scheduler involves assessing its ability to allocate resources efficiently, handle task prioritization correctly, and prevent resource contention or priority inversion scenarios.
Furthermore, latency measurement is a critical part of Real-Time Linux testing. Latency refers to the time delay between the occurrence of an event and the system's response to it. In real-time applications, minimizing latency is crucial to achieving timely and predictable behavior. Testing latency involves measuring the time it takes for the system to respond to various stimuli and identifying any sources of delay or unpredictability.
Additionally, stress testing plays a significant role in assessing the system's robustness under heavy workloads. It involves subjecting the Real-Time Linux system to high levels of concurrent activities, intense computational loads, and input/output operations to evaluate its performance, responsiveness, and stability. Stress testing helps identify potential bottlenecks, resource limitations, or issues that might degrade the real-time behavior of the system.

##### RTLA – The realtime Linux analysis tool

    RTLA – The realtime Linux analysis tool:
       rtla timerlat$_{doc}$ – CLI for the kernel's timerlat tracer$_{doc}$
       rtla osnoise$_{doc}$ – CLI for the kernel's osnoise tracer$_{doc}$.
          Kernel function run_osnoise$_{id}$ measures time with function trace_clock_local$_{id}$ in loop.
       rtla hwnoise$_{doc}$ – CLI for the osnoise tracer$_{doc}$ with interrupts disabled
          Implementation: tools/tracing/rtla$_{src}$ and kernel/trace/trace_osnoise.c$_{src}$
       Linux scheduling latency debug and analysis

##### RT-Tests

    RT-Tests, source
       cyclictest
    some RT-Tests man pages:
       cyclictest – measures man 2 clock_nanosleep or man 2 nanosleep delay
       hwlatdetect – CLI for /sys/kernel/tracing/hwlat_detector$_{doc}$ / kernel/trace/trace_hwlat.c$_{src}$. Kernel function kthread_fn$_{id}$ measures time delays with function trace_clock_local$_{id}$ in loop.
       oslat – measures delay with RDTSC in busy loop
       hackbench – scheduler benchmark/stress test

##### Other tests

    tools/tracing/latency$_{src}$
    CONFIG_IRQSOFF_TRACER$_{id}$ – interrupts-off latency tracer
    CONFIG_PREEMPT_TRACER$_{id}$ – preemption-off latency tracer
    CONFIG_SCHED_TRACER$_{id}$ – scheduling latency tracer
    RT Tracing Tools with eBPF
    realtime$_{ltp}$
    https://www.latencytop.org/, kernel/latencytop.c$_{src}$
Further reading about real-time Linux:
    linux/spinlock_rt.h$_{inc}$ used via linux/spinlock_types.h$_{inc}$
    linux/rtmutex.h$_{inc}$ used via linux/mutex_types.h$_{inc}$
    linux/rwbase_rt.h$_{inc}$ used via linux/rwlock_types.h$_{inc}$
    Introduction to Real-Time Linux: Unleashing Deterministic Computing
    Power Management and Scheduling in the Linux Kernel (OSPM)
    the Real-Time Linux wiki
       CPU partitioning and isolation
    Realtime@LWN
    linux-stable-rt.git
    linux-rt-devel.git
    Realtime kernel patchset, Arch Linux
    https://www.kernel.org/pub/linux/kernel/projects/rt/ - RT patches for upstream kernel
    High Precision Event Timer (HPET)
    Demystifying the Real-Time Linux Scheduling Latency
    Real-time kernel tuning in RHEL 9
    Linux subsystems related to real-time
       Linux kernel scheduling and preemption
       Interrupts
       Deferred works
       Non-maskable interrupt handler (NMI)
       System management interrupt (SMI)
       man 7 sched
    latency @ LKML
    PREEMPT_RT @ LKML
    QA about PREEMP_RT, LPC'23, State of the onion, pdf
💾 Historical
The PREEMPT_RT patch has been partially merged into the mainline Linux kernel, starting from version 5.15.