# The Linux Kernel/Debugging
## Performance

There are many factors that can affect the performance of the Linux kernel, including hardware configurations, software configurations, and workload characteristics.
In this context, performance optimization of the Linux kernel involves identifying and addressing performance bottlenecks in the system. This can involve tuning kernel parameters, optimizing system resources, and identifying and fixing bugs and other issues that may be impacting performance.
Given the complexity of the Linux kernel and the wide range of factors that can affect performance, performance optimization can be a challenging task. However, with the right tools and techniques, it is possible to significantly improve the performance and reliability of Linux-based systems.

### Perf_events

Perf_events, short for performance events, is a powerful interface that provides detailed insights into the performance characteristics of software running on a system. By analyzing the data collected by perf_events, developers can identify performance bottlenecks and optimize software to improve performance and reduce resource utilization. Perf_events is designed to be a lightweight, low-overhead monitoring solution that has minimal impact on system performance.
🔧 TODO
⚲ Interfaces
    man 1 perf – performance analysis tools
    Basic commands:
       man 1 perf-help – display help information about perf
       man 1 perf-top – System profiling tool.
       man 1 perf-record – Run a command and record its profile into perf.data
       man 1 perf-report – Read perf.data (created by perf record) and display the profile
    Other commands:
       man 1 perf-annotate – Read perf.data (created by perf record) and display annotated code
       man 1 perf-archive – Create archive with object files with build-ids found ...
       man 1 perf-arm-spe – Support for Arm Statistical Profiling Extension within...
       man 1 perf-bench – General framework for benchmark suites
       man 1 perf-buildid-cache – Manage build-id cache.
       man 1 perf-buildid-list – List the buildids in a perf.data file
       man 1 perf-c2c – Shared Data C2C/HITM Analyzer.
       man 1 perf-config – Get and set variables in a configuration file.
       man 1 perf-daemon – Run record sessions on background
       man 1 perf-data – Data file related processing
       man 1 perf-diff – Read perf.data files and display the differential profile
       man 1 perf-dlfilter – Filter sample events using a dynamically loaded shared...
       man 1 perf-evlist – List the event names in a perf.data file
       man 1 perf-ftrace – simple wrapper for kernel's ftrace functionality
       man 1 perf-inject – Filter to augment the events stream with additional in...
       man 1 perf-intel-pt – Support for Intel Processor Trace within perf tools
       man 1 perf-iostat – Show I/O performance metrics
       man 1 perf-kallsyms – Searches running kernel for symbols
       man 1 perf-kmem – Tool to trace/measure kernel memory properties
       man 1 perf-kvm – Tool to trace/measure kvm guest os
       man 1 perf-kwork – Tool to trace/measure kernel work properties (latencies)
       man 1 perf-list – List all symbolic event types
       man 1 perf-lock – Analyze lock events
       man 1 perf-mem – Profile memory accesses
       man 1 perf-probe – Define new dynamic tracepoints
       man 1 perf-sched – Tool to trace/measure scheduler properties (latencies)
       man 1 perf-script – Read perf.data (created by perf record) and display tr...
          man 1 perf-script-perl – Process trace data with a Perl script
          man 1 perf-script-python – Process trace data with a Python script
       man 1 perf-stat – Run a command and gather performance counter statistics
       man 1 perf-test – Runs sanity tests.
       man 1 perf-timechart – Tool to visualize total system behavior during a workload
       man 1 perf-trace – strace inspired tool
       man 1 perf-version – display the version of perf binary
⚙️ Internals
    man 2 perf_event_open – sets up performance monitoring
    uapi/linux/perf_event.h$_{inc}$
    tools/perf$_{src}$
    linux/perf_event.h$_{inc}$
    kernel/events/core.c$_{src}$
    kernel/profile.c$_{src}$ – simple profiling
📖 References
    perf – instruments CPU performance counters, tracepoints, kprobes, and uprobes
    https://perf.wiki.kernel.org/
📚 Further reading
    perf Examples
    The Unofficial Linux Perf Events Web-Page
🛠️ Utilities
    Performance Co-Pilot, https://pcp.io/ – Performance Co-Pilot
    Prometheus, https://prometheus.io/
    https://github.com/redhat-nfvpe/container-perf-tools
    https://github.com/brendangregg/perf-tools – performance analysis tools based on Linux perf_events (aka perf) and ftrace
    readprofile – a tool to read kernel profiling information
📚 Further reading
    stress-ng – exercises various kernel interfaces
    http://trac.gateworks.com/wiki/linux/profiling
    Analyzing application performance in RHEL 9
    Monitoring and managing system status and performance in RHEL 9
    Real-time Linux

## User space debug interfaces

⚲ Interfaces
    man 1 dmesg – prints or control the kernel ring buffer
       man 2 syslog – system call, which is used to control the kernel printk buffer
    man 1 strace – system calls and signals tracing tool
       man 2 ptrace – process trace system call
    man 3 klogctl
    man 5 core
    /sys/kernel/debug/ – debugfs
    dmesg --console-level <level>
    gdb /usr/src/linux/vmlinux /proc/kcore
    /proc/self/stack
    dynamic$_{doc}$ debug
       ⌨️ hands-on:
       echo "module atkbd +pfl" | sudo tee /sys/kernel/debug/dynamic_debug/control
⚙️ Internals
    handle_sysrq$_{id}$
📚 References
    Development tools for the kernel$_{doc}$
    DebugFS$_{doc}$, samples/qmi/qmi_sample_client.c$_{src}$
    Kprobe-based Event Tracing$_{doc}$
    Dynamic debug$_{doc}$
    Linux Magic System Request Key Hacks$_{doc}$
    Magic SysRq key

## Tracing and logging

⚲ API:
User-space interface:
    man 1 dmesg – prints or control the kernel ring buffer
       man 2 syslog – system call, which is used to control the kernel printk buffer
    /proc/kmsg
    https://kernelshark.org/ – front end reader of trace-cmd
    https://trace-cmd.org/, man 1 trace-cmd – CLI for Ftrace$_{doc}$ – Linux kernel internal tracer /sys/kernel/debug/tracing/
Most common functions
    linux/printk.h$_{inc}$
       pr_devel$_{id}$- conditional debug-level message
       pr_debug$_{id}$- conditional debug-level or dynamic$_{doc}$ message
          ⌨️ hands-on:
          echo "module atkbd +pfl" | sudo tee /sys/kernel/debug/dynamic_debug/control
       Log messages with other levels:
          pr_info$_{id}$, pr_notice$_{id}$, pr_warn$_{id}$, pr_err$_{id}$, pr_crit$_{id}$, pr_alert$_{id}$, pr_emerg$_{id}$
    asm-generic/bug.h$_{inc}$
       WARN_ON$_{id}$
       WARN$_{id}$
⚙️ Internals
    printk$_{id}$
    kernel/printk/printk.c$_{src}$
    arch/x86/kernel/traps.c$_{src}$
    lib/dump_stack.c$_{src}$
    kernel/trace$_{src}$
    scripts/tracing/draw_functrace.py$_{src}$
    logging$_{ltp}$, tracing$_{ltp}$
    samples/ftrace$_{src}$
    samples/trace_events$_{src}$
    samples/trace_printk$_{src}$
    linux/instrumentation.h$_{inc}$
📚 References:
    Debugging by printing
    Message logging with printk$_{doc}$
    SystemTap
       man 1 stap – systemtap script translator/driver
    strace
       man 1 strace – trace system calls and signals
    LTTng
    ftrace
    Linux Tracing Technologies$_{doc}$
       Tracepoint Analysis$_{doc}$
       Function Tracer$_{doc}$ – function, latency and event tracing
          Event Tracing$_{doc}$
       Using ftrace to hook to functions$_{doc}$
       Fprobe - Function entry/exit probe$_{doc}$
       Kprobes$_{doc}$
       Kprobe-based Event Tracing$_{doc}$
       Uprobe-tracer: Uprobe-based Event Tracing$_{doc}$
       Using the Linux Kernel Tracepoints$_{doc}$
       Subsystem Trace Points: kmem$_{doc}$
       Subsystem Trace Points: power$_{doc}$
       NMI Trace Events$_{doc}$
       In-kernel memory-mapped I/O tracing$_{doc}$
       Event Histograms$_{doc}$
       Histogram Design Notes$_{doc}$
       Boot-time tracing$_{doc}$
       Hardware Latency Detector$_{doc}$
       Intel(R) Trace Hub (TH)$_{doc}$
       Lockless Ring Buffer Design$_{doc}$
       System Trace Module$_{doc}$
       CoreSight - ARM Hardware Trace$_{doc}$
🔧 TODO. 🚀 advanced features
    linux/kmemleak.h$_{inc}$ – memory leak detector
    pr_cont$_{id}$- continues a previous log message in the same line
    print_hex_dump_bytes$_{id}$
    print_hex_dump_debug$_{id}$
    dump_stack$_{id}$
    CONFIG_PRINTK_CALLER$_{id}$
    CONFIG_DEBUG_KERNEL$_{id}$
    CONFIG_DEBUG_INFO$_{id}$
    https://git.kernel.org/pub/scm/libs/libtrace/

## kgdb and kdb

⚲ Interfaces
    linux/kgdb.h$_{inc}$
    linux/kdb.h$_{inc}$
⚙️ Internals
    kernel/debug$_{src}$
📚 References
    Using kgdb, kdb and the kernel debugger internals$_{doc}$
    kdump
    kdump$_{doc}$
    man 8 crash – Analyze Linux crash dump data or a live system

## eBPF

⚲ API:
    man 2 bpf ↪ kernel/bpf/syscall.c$_{src}$
📖 References
    eBPF and BPF$_{doc}$
📚 Further reading
    man 7 bpf-helpers
    Linux Extended BPF (eBPF) Tracing Tools
    bpftrace – High-level tracing language for Linux eBPF
    BCC – Tools for BPF-based Linux IO analysis, networking, monitoring, and more
       Example of trace.py
    man 8 stapbpf
    eBPF Programming for Linux Kernel Tracing
    lockdep - Runtime locking correctness validator$_{doc}$

## Watchdogs

The Linux Kernel/Softdog Driver
dev_watchdog$_{id}$ – network device watchdog
The NMI watchdog lockup detectors:
⚲ API
    /proc/sys/kernel/nmi_watchdog
    /proc/sys/kernel/soft_watchdog
    /proc/sys/kernel/watchdog
    /proc/sys/kernel/watchdog_cpumask
    /proc/sys/kernel/watchdog_thresh
    /proc/sys/kernel/hardlockup_all_cpu_backtrace
    /proc/sys/kernel/hardlockup_panic
    /proc/sys/kernel/softlockup_all_cpu_backtrace
    /proc/sys/kernel/softlockup_panic
    linux/nmi.h$_{inc}$
👁️ Example
    ./lib/test_lockup.c$_{src}$ – test module to generate lockups
Provoke NMI watchdog without panic:
⚙️ Internals
    kernel/watchdog.c$_{src}$ – detects hard and soft lockups on a system
    kernel/watchdog_perf.c$_{src}$ – detects hard lockups on a system using perf
    kernel/watchdog_buddy.c$_{src}$
📚 References
    Documentation for /proc/sys/kernel/$_{doc}$
    Softlockup detector and hardlockup detector (aka nmi_watchdog)$_{doc}$
    kernel parameters:
       nmi_watchdog$_{param}$
       nowatchdog$_{param}$
       nosoftlockup$_{param}$
       softlockup_panic$_{param}$
⚙️ Internals
    arch/x86/kernel/traps.c$_{src}$
📖 References for debugging
    Ramoops oops/panic logger$_{doc}$
    pstore block oops/panic logger$_{doc}$
    Fault injection$_{doc}$
    Bisecting a bug$_{doc}$
    Development tools for the kernel$_{doc}$
       Kernel Testing Guide$_{doc}$
       Checkpatch$_{doc}$, scripts/checkpatch.pl$_{src}$
       Selftests$_{doc}$, tools/testing/selftests$_{src}$
    linux/tracepoint.h$_{inc}$
📚 Further reading
    https://drgn.readthedocs.io/ – programmable debugger
    https://crash-utility.github.io/
    https://wiki.ubuntu.com/Kernel/Debugging
    Linux Applications Debugging Techniques