# Embedded Systems/Linux

Embedded Linux systems can range from consumer electronics like smartphones and smart TVs to industrial machinery, medical devices, automotive systems, communication equipment and more. Linux brings the power, flexibility, and stability to these resource-constrained devices.
Core components of an Linux on embedded systems are boot loader, custom Linux kernel and root file system. Source code is compiled by GCC cross compiler. Common boot loader is Das U-Boot – the Universal Boot Loader. Root FS usually is based on BusyBox – software suite that provides many Unix utilities in a single executable file. Local debugging often is limited, so gdbserver can be used.
Basic embedded Linux system can be compiled manually from modules above. There are tools for complete system generation from old and simple Buildroot to huge Yocto Project, which creates customized embedded Linux distributions. PC distributions Debian, Ubuntu ant others have build for ARM processors.
Raspberry Pi, BeagleBoard and Nvidia Jetson are examples of popular single-board computers supporting embedded Linux.
Unlike regular desktop or server distributions of Linux these versions of Linux usually have much fewer libraries and application, firmware is smaller and typically boot from flash memory or SD card.

## Projects
### Build frameworks and environments

    Wear OS – Android for smartwatches and other wearables.
    AOSP – Android Open Source Project
    Buildroot – old simple patch and make based build environment
    Yocto Project and OpenEmbedded – a BitBake based build framework and environment
    BitBake – make-like build tool
    OpenWrt – open wireless router
    Embeddable Linux Kernel Subset – for ancient 16-bit x86 machines and emulators

### Components

    The Linux Kernel
    "init" implementations: OpenRC, runit
    GUIs: LXQt, LXDE, Xfce, FLTK
    coreboot – lightweight BIOS

#### C standard library variants

    General purpose GNU C Library glibc, 7.9 MB
    uClibc, 560 KB
    musl, 527 KB
    dietlibc, 185 KB
    Newlib
    klibc – primarily for booting Linux systems
    Bionic – originally developed by Google for the Android embedded system operating system
Further reading:
    Comparison of C/POSIX standard library implementations for Linux
    Choosing System C library

## Distributions

Distributions for ARM processors:
    Arch Linux ARM – port of Arch Linux for ARM processors
    Armbian – Debian and Ubuntu based
    RedSleeve – port of Red Hat Enterprise Linux
    emteria.OS – proprietary Android based
Lightweight cloud oriented systems:
    Alpine Linux – uses musl, BusyBox and OpenRC.
    Fedora CoreOS
    RHCOS – Red Hat Enterprise Linux CoreOS for OpenShift Container Platform
    https://MicroShift.io/ – lightweight Kubernetes distribution designed for edge and IoT computing environments
Lightweight multi-platform distributions:
    Gentoo Linux
    Tiny Core Linux
    Kali Linux
    Lubuntu
    Slackware
    Void Linux
    Xubuntu
Common multi-platform distributions:
    Debian
    Fedora
    openSUSE
    Ubuntu

## Features of Linux kernel for embedded systems

    I/O and buses:
       GPIO
          drivers/gpio$_{src}$, tools/gpio$_{src}$, General Purpose Input/Output$_{doc}$
       SPI
          drivers/spi$_{src}$, tools/spi$_{src}$, Serial Peripheral Interface$_{doc}$
       I$^2$C
          ⚲ UAPI: Instantiate I2C devices from user-space$_{doc}$
          drivers/i2c$_{src}$, https://i2c.wiki.kernel.org
          I2C/SMBus Subsystem$_{doc}$
       drivers/net/can$_{src}$ – Controller Area Network bus
       drivers/mfd$_{src}$ – Multi-Function Devices
    drivers/usb/gadget$_{src}$ – peripheral USB device implementation
    drivers/iio$_{src}$ – Industrial I/O
    drivers/pwm$_{src}$ – Pulse-width modulation
    drivers/regulator$_{src}$ – Generic Voltage and Current Regulator support
    drivers/fpga$_{src}$
    drivers/slimbus$_{src}$
    SLOB – Simple List Of Blocks memory allocator
    Devicetree – describes hardware components for the kernel

### Multimedia

    Audio
       ASoC – ALSA System on Chip
       snd_soc_card$_{id}$
       ASoC – ALSA SoC Layer$_{doc}$
       https://www.alsa-project.org/wiki/ASoC
    TFT (LCD) – Thin-film-transistor liquid-crystal display
       fbtft_display$_{id}$

### File systems and storage

    UBIFS – Unsorted Block Image FS
    SquashFS – a compressed read-only FS with low overhead
    drivers/mtd$_{src}$ – Memory Technology Device block device
    More: YAFFS, JFFS2

### MMU less

Linux was originally designed on a processor with a memory management unit (MMU). Most embedded systems do not have a MMU, as we discussed earlier (Embedded Systems/Memory) and are called microcontrollers. Microcontrollers without a MMU are cheaper consumes less power.
Benefits of using a processor with a MMU:
    can isolate running "untrusted" machine code from running "critical" code, so the "untrusted" code is guaranteed (in the absence of hardware failures) not to interfere with the "critical" code
    makes it easier for the OS to present the illusion of virtual memory
    can run "normal" Linux (could also run "μClinux", but what's the point?)
μClinux – "MicroController Linux" is a version of the Linux kernel that supports Altera NIOS, ARM, Freescale M68K (including DragonBall, ColdFire, PowerQUICC and others), Hitachi H8, MIPS, and Xilinx MicroBlaze processors.
Disabling parameter CONFIG_MMU enables nommu mode.

### Real-time

People use a variety of methods to combine real-time tasks with Linux:
* Run the real-time tasks on a dedicated microcontroller; communicate with a (non-real-time) PC that handles non-real-time tasks. This is pretty much your only choice if you need real-time response times below 1 microsecond.
* Run the real-time tasks in a "underlying" dedicated real-time operating system; run Linux as a "nested operating system" inside one low-priority task on top of the real-time operating system. Some of these systems claim real-time response times below 500 microseconds.
* use a Linux kernel designed to emphasize real-time tasks, and run the real-time tasks with a high priority (perhaps even as a kernel thread).
Examples of RT distributions:
    Xenomai
    RTAI
    RTLinux
    MontaVista Linux
    Red Hat Enterprise Linux for Real Time
    Real-time Ubuntu
Several RT features provide real time capabilities:
    RT preemption
    RT scheduling
    CPU partitioning and isolation
    Dynticks (Tickless, nohz)

#### RT preemption

The PREEMPT_RT patch has been partially merged into the mainline Linux kernel, starting from version 5.15. This means that real-time capabilities are now available as part of the mainline Linux kernel, making it easier to deploy and maintain real-time systems using Linux.
Lazy preemption (CONFIG_PREEMPT_LAZY) remains in the external patchset.
However, the real-time capabilities of Linux are still evolving, and there are ongoing efforts to improve real-time performance and reduce latency even further. The Linux Foundation's Real-Time Linux (RTL) collaborative project is focused on improving the real-time capabilities of Linux and advancing the adoption of real-time Linux in various industries, including aerospace, automotive, robotics, and telecommunications.
In summary, real-time Linux is now part of the mainline Linux kernel, and there are ongoing efforts to improve real-time performance and reduce latency even further.
Parameter CONFIG_PREEMPT_RT$_{id}$ enables real-time preemption.

#### RT scheduling policies

Scheduling policies for RT:
    SCHED_FIFO$_{id}$, SCHED_RR$_{id}$
       implemented in kernel/sched/rt.c$_{src}$
    SCHED_DEADLINE
       implemented in kernel/sched/deadline.c$_{src}$
API:
    man 1 chrt – manipulate the real-time attributes of a process
    man 2 sched_rr_get_interval – get the SCHED_RR interval for the named process
    man 2 sched_setscheduler, sched_getscheduler – set and get scheduling policy/parameters
    man 2 sched_get_priority_min, sched_get_priority_max – get static priority range

#### Testing RT capabilities

The testing process for Real-Time Linux typically involves several key aspects. First and foremost, it is crucial to verify the accuracy and stability of the system's timekeeping mechanisms. Precise time management is fundamental to real-time applications, and any inaccuracies can lead to timing errors and compromise the system's real-time capabilities.
Another essential aspect of testing is evaluating the system's scheduling algorithms. Real-Time Linux employs advanced scheduling policies to prioritize critical tasks and ensure their timely execution. Testing the scheduler involves assessing its ability to allocate resources efficiently, handle task prioritization correctly, and prevent resource contention or priority inversion scenarios.
Furthermore, latency measurement is a critical part of Real-Time Linux testing. Latency refers to the time delay between the occurrence of an event and the system's response to it. In real-time applications, minimizing latency is crucial to achieving timely and predictable behavior. Testing latency involves measuring the time it takes for the system to respond to various stimuli and identifying any sources of delay or unpredictability.
Additionally, stress testing plays a significant role in assessing the system's robustness under heavy workloads. It involves subjecting the Real-Time Linux system to high levels of concurrent activities, intense computational loads, and input/output operations to evaluate its performance, responsiveness, and stability. Stress testing helps identify potential bottlenecks, resource limitations, or issues that might degrade the real-time behavior of the system.
    RTLA – The realtime Linux analysis tool:
       rtla timerlat$_{doc}$ – CLI for the kernel's timerlat tracer$_{doc}$
       rtla osnoise$_{doc}$ – CLI for the kernel's osnoise tracer$_{doc}$. Kernel function run_osnoise$_{id}$ measures time with function trace_clock_local$_{id}$ in loop.
       rtla hwnoise$_{doc}$ – CLI for the osnoise tracer$_{doc}$ with interrupts disabled
       Implementation: tools/tracing/rtla$_{src}$ and kernel/trace/trace_osnoise.c$_{src}$
       Linux scheduling latency debug and analysis
    RT-Tests, source
       cyclictest
    some RT-Tests man pages:
       cyclictest – measures man 2 clock_nanosleep or man 2 nanosleep delay
       hackbench – scheduler benchmark/stress test
       hwlatdetect – CLI for /sys/kernel/tracing/hwlat_detector$_{doc}$ / kernel/trace/trace_hwlat.c$_{src}$. Kernel function kthread_fn$_{id}$ measures time delays with function trace_clock_local$_{id}$ in loop.
       oslat – measures delay with RDTSC in busy loop
    RT Tracing Tools with eBPF
    realtime$_{ltp}$
Further reading about real-time Linux:
    linux/spinlock_rt.h$_{inc}$ used via linux/spinlock_types.h$_{inc}$
    linux/rtmutex.h$_{inc}$ used via linux/mutex_types.h$_{inc}$
    linux/rwbase_rt.h$_{inc}$ used via linux/rwlock_types.h$_{inc}$
    Introduction to Real-Time Linux: Unleashing Deterministic Computing
    Power Management and Scheduling in the Linux Kernel (OSPM)
    the Real-Time Linux wiki
       CPU partitioning and isolation
    Realtime@LWN
    linux-stable-rt.git
    linux-rt-devel.git
    Realtime kernel patchset, Arch Linux
    https://www.kernel.org/pub/linux/kernel/projects/rt/ - RT patches for upstream kernel
    High Precision Event Timer (HPET)
    Demystifying the Real-Time Linux Scheduling Latency
    Real-time kernel tuning in RHEL 9
    Linux subsystems related to real-time
       Linux kernel scheduling and preemption
       Interrupts
       Deferred works
       Non-maskable interrupt handler (NMI)
       System management interrupt (SMI)
       man 7 sched
    latency @ LKML
    PREEMPT_RT @ LKML
    QA about PREEMP_RT, LPC'23, State of the onion, pdf
Further readings about Embedded Linux:
    Embedded Linux Conference (ELC) @ EOSS 2023
    https://elinux.org
    Category:Embedded_Linux
    Linux kernel interfaces
    ARM support
       arch/arm$_{src}$, ARM Architecture$_{doc}$
       https://lwn.net/Kernel/Index/#Architectures-ARM
       arch/arm64$_{src}$, ARM64 Architecture$_{doc}$
    architecture-specific initialization
    Learning the vi Editor
* ↑ "Realtime preemption locking core merged". {{cite journal}}: Cite journal requires |journal= (help)