Falco Runtime Security: 20251205 multi thread falco design

# Multi-Threaded Falco High-Level Design (Working draft)

## Summary

This document outlines a high-level design for implementing multi-threading in Falco. The goal of this proposal is to overcome Falco's single-threaded architecture to improve scalability in scenarios where the amount of events produced cannot be processed in a single thread. This is achieved by leveraging multiple threads for event processing, rule evaluation, and output handling, enabling Falco to better utilize multi-core systems and reduce event drops under high event rates.

## Goals

* Address the problems related to single CPU core saturation, leading to dropped events.
* Minimize the performance impact on the single threaded usage, that remains the default. 

## Non-Goals

* This document does not cover low-level implementation details that will be addressed in specific design documents for each component or directly in the implementation phase.
* This document does not focus on performance optimization, the primary goal is scalability improvements to handle higher event rates that exceed single-thread processing capacity.

## Success Metrics

The success of this multi-threading initiative will be measured by the following key metrics:

* **Event Drop Rate Reduction**: The primary success metric is the reduction in event drops under high event rates. A successful implementation should significantly reduce or eliminate event drops that occur when a single thread cannot keep up with the event rate.
* **Throughput Scaling**: The system should demonstrate improved throughput (events processed per second) that scales with the number of worker threads, up to a reasonable limit based on available CPU cores and workload characteristics.
* **CPU Utilization**: Multi-threaded Falco should better utilize available CPU cores, with worker threads distributing load across multiple cores instead of saturating a single core.
* **Single-Threaded Performance Preservation**: The single-threaded mode (default) should maintain its current performance characteristics, with minimal or no performance regression when multi-threading is disabled.

These metrics will be evaluated through benchmarking and real-world deployment scenarios to validate that the multi-threaded architecture achieves its scalability goals without compromising correctness or introducing significant overhead.

## High-Level Design

### Current Architecture

![Current Falco Architecture](images/falco-architecture.png)

* The kernel driver (via kmod or eBPF) writes events into per-CPU ring buffers. Each CPU has its own buffer to avoid lock contention. We have a ring buffer per CPU core, and a single userspace.
* Userspace (libscap) performs an `O(n_cpus)` scan on every next() call, it peeks at the head event from each ring buffer, finds the event with the minimum timestamp across all buffers and returns that event to Falco for processing. The consumer position is only advanced after the event has been consumed (on the next call), ensuring the caller can safely read the event data and avoiding the need to perform copies of the event data.
* Libsinsp processes the events sequentially as they are received from libscap, building a stateful representation of the system and providing the necessary context for rule evaluation.
* Falco evaluates the rules against the processed events and generates alerts based on the defined security policies.

### Proposed Architecture Overview

![Multi-Threaded Falco Architecture](images/falco-multi-thread-architecture.png)

* The kernel driver (modern eBPF probe) routes events into per-partition ring buffers based on TGID. The routing logic executes in kernel space (within the eBPF program), where each event's TGID is hashed and used to select the target ring buffer. Only the modern eBPF probe is supported, as it relies on [BPF_MAP_TYPE_RINGBUF](https://docs.ebpf.io/linux/map-type/BPF_MAP_TYPE_RINGBUF/) which does not have a per-CPU design as opposed to the `BPF_MAP_TYPE_PERF_EVENT_ARRAY` used by the legacy eBPF probe.
* Each ring buffer is associated with an event loop worker thread that processes events from its assigned ring buffer.
* The `libsinsp` state (e.g., the thread state) is maintained in a shared data structure, allowing all workers to access data pushed by other workers. This is crucial for handling events like clone() that rely on data written by other partitions. This requires designing lightweight synchronization mechanisms to ensure efficient access to shared state without introducing significant contention. A dedicated proposal document will address the design of the shared state and synchronization mechanisms, and data consistency.
* Falco's rule evaluation is performed in parallel by multiple worker threads, each evaluating rules against the events they process. Current Falco plugins are not supposed to be thread-safe. A dedicated proposal document will address the design of a thread-safe plugin architecture.
* **Output handling** is already designed for multi-threaded access. The `falco_outputs` class implements a thread-safe, queue-based architecture using Intel TBB's `concurrent_bounded_queue`, which is specifically designed for multi-producer, single-consumer scenarios. Multiple worker threads can concurrently call `handle_event()` to enqueue alert messages using the thread-safe `try_push()` operation. A dedicated output worker thread consumes messages from the queue using `pop()` and sends them to all configured outputs (stdout, file, syslog, gRPC, HTTP, etc.). This design is already proven in production, as Falco's multi-source support (where different event sources run in separate threads) already uses this same queue concurrently. The existing implementation requires no changes to support multi-threaded event processing. Note that while outputs are processed in order within the queue, alerts from different worker threads may be interleaved, meaning strict temporal ordering of alerts across different processes is not guaranteed. This is acceptable for security monitoring use cases where the primary concern is detecting and reporting security events rather than maintaining precise event ordering.

### Work Partitioning Strategies

A crucial and challenging design aspect is partitioning the work to achieve a good trade-off among the following properties:

1. **Even load balancing** between threads
2. **Low contention** on shared data (or no shared data)
3. **Avoiding temporal inconsistencies and causality violations** (e.g., processing a file-opening event before the related process-forking event)

The first two properties are primarily focused on performance, while the third is essential for the correctness of the solution. These aspects are intrinsically linked.

Based on the analysis below, **Static Partitioning by TGID** is the proposed approach for the initial implementation.

#### Static Partitioning by TGID (Thread Group ID / Process ID)

Events are routed based on the TGID in kernel spa