GraphBit's Multi-Core AI Workflows: Achieving True Parallel Processing with Independent Node Execution

Musa Molla

Musamolla

Follow

Executive Overview

GraphBit executes AI workflows as graphs of independent nodes scheduled across a true multi-threaded runtime. Instead of coordinating coroutines in a single interpreter thread, GraphBit maps ready nodes to worker threads that run in parallel on multiple CPU cores, delivering multi-core throughput for mixed I/O + CPU workloads typical of LLM systems.

Execution Model

• Workflow as DAG: Nodes declare dependencies; any set of ready nodes forms a batch.
• Batching policy: create_execution_batches(..) chunks ready nodes up to max_concurrency(); batches are fired concurrently.
• Parallel futures: Node execution is driven with join_all(futures), allowing the Tokio multi-thread scheduler to place each future on a separate worker thread.
• Separation of concerns: CPU-bound work (token prep, parsing, light transforms) runs on worker threads; I/O (HTTP to providers, file/db calls) is moved to a dedicated blocking pool.

Runtime & Threading

• Thread pool sizing: Default worker_threads = clamp(2× CPU cores, 4…32) balances I/O waits and CPU bursts while containing context-switch overhead.
• Blocking pool: max_blocking_threads = 4× CPU cores prevents I/O waits from starving compute threads.
• Stack tuning: 1 MB per thread is chosen to minimize resident footprint without constraining typical call depths.

Why This Delivers Real Parallelism

• Rust core, no GIL: Tokio’s Builder::new_multi_thread() runs truly in parallel across cores.
• Safety without locks: Rust’s ownership model eliminates data races without introducing global locks, reducing contention hotspots.
• Deterministic boundaries: Concurrency is explicit at the node and batch level, making behavior predictable and testable.

Observability & Control

• System introspection: API exposes CPU count, configured worker threads, allocator choice, and runtime status—useful for capacity planning.
• CPU affinity (optional): Aligns worker threads with pinned cores when hosts enforce affinity, limiting cross-core thrash.

Failure Semantics

• Isolated node failures: A node failing in one batch does not poison others; retries and circuit breakers operate at node/request granularity.
• Backpressure: If provider or downstream services slow, batch sizing and concurrency are throttled to prevent queue explosion.

Trade-offs

• Upper bound at 32 workers: Prevents diminishing returns from rampant context switching on mainstream hardware; can be made configurable for high-core hosts.
• Batch sizing vs latency: Larger batches improve throughput but can increase tail latency—tune per workflow.

Practical Guidance

• For independent fan-out graphs (retrieval + multi-tool analyses), keep max_concurrency() near the default.
• For long chains with heavy per-node CPU transforms, consider nudging worker_threads above default (within cap).
• Keep nodes pure and idempotent; it simplifies retries and parallel scheduling.