Repository: oqtopus-team/oqtopus-engine
Pull request #13: feat: Add Config-Driven Pipeline Construction via PipelineBuilder
Author: login: snuffkin
Created: 2026-02-27T01:07:36Z | Merged: 2026-02-27T01:07:47Z

## ✍ Description

This PR introduces **configuration-driven pipeline construction** to the OQTOPUS Engine Core.  
Pipeline definitions no longer need to be hard-coded in `app.py`.  
The new `PipelineBuilder` creates a `PipelineExecutor` based on `pipeline_executor` settings defined in `config.yaml` and components registered in the DI container.

This improves flexibility, clarity, and maintainability across the engine.

Key Changes:

- Added `PipelineBuilder` to construct the pipeline from configuration.
- `app.py` now delegates pipeline creation to `PipelineBuilder` instead of instantiating steps manually.
- Added documentation section **“2. Pipeline Construction from `config.yaml`”** explaining the configuration-driven model.
- Updated tests to include `PipelineBuilder` unit tests.


Benefits:

- Enables **declarative** pipeline specification.
- Clean separation between:
  - pipeline **construction** (configuration & DI),
  - pipeline **execution** (PipelineExecutor traversal).
- Simplifies modification or reordering of steps without touching code.
- Easier to manage environment-specific pipeline configurations.


Notes:

- This change does **not** alter pipeline execution semantics.
- Component names in `pipeline_executor.pipeline` must match keys in `di_container.registry`.
- For full configuration details, refer to the updated documentation.


Diff:
diff --git a/core/config/config.yaml b/core/config/config.yaml
index 8805e8f..15e93e2 100644
--- a/core/config/config.yaml
+++ b/core/config/config.yaml
@@ -1,3 +1,18 @@
+# Pipeline Executor Configuration (get instances from DI container)
+pipeline_executor:
+  pipeline:
+    - job_repository_update_step
+    - multi_manual_step
+    - tranqu_step
+    - estimator_step
+    - ro_error_mitigation_step
+    - buffer
+    - sse_step
+    - device_gateway_step
+  job_buffer: buffer
+  exception_handler: exception_handler
+
+# Dependency Injection Container Configuration
 di_container:
   registry:
     # fetcher configurations
diff --git a/core/config/sse_engine_config.yaml b/core/config/sse_engine_config.yaml
index 0682acb..57f51a8 100644
--- a/core/config/sse_engine_config.yaml
+++ b/core/config/sse_engine_config.yaml
@@ -1,3 +1,18 @@
+# Pipeline Executor Configuration (get instances from DI container)
+pipeline_executor:
+  pipeline:
+    - job_repository_update_step
+    - multi_manual_step
+    - tranqu_step
+    - estimator_step
+    - ro_error_mitigation_step
+    - buffer
+    - sse_step
+    - device_gateway_step
+  job_buffer: buffer
+  exception_handler: pipeline_exception_handler
+
+# Dependency Injection Container Configuration
 di_container:
   registry:
     # fetcher configurations
diff --git a/core/src/oqtopus_engine_core/app.py b/core/src/oqtopus_engine_core/app.py
index 72ef753..5f1c9d4 100644
--- a/core/src/oqtopus_engine_core/app.py
+++ b/core/src/oqtopus_engine_core/app.py
@@ -2,13 +2,11 @@
 import logging
 
 from oqtopus_engine_core.framework import (
-    Buffer,
     DeviceFetcher,
     GlobalContext,
     JobFetcher,
     JobRepository,
-    PipelineExceptionHandler,
-    PipelineExecutor,
+    PipelineBuilder,
 )
 from oqtopus_engine_core.utils import (
     load_config,
@@ -38,28 +36,8 @@ async def main() -> None:
     # Initialize the DI container
     dicon = DiContainer(gctx.config["di_container"]["registry"])
 
-    # Initialize the pipeline exception handler
-    exception_handler: PipelineExceptionHandler = dicon.get(
-        "pipeline_exception_handler"
-    )
-
-    # Initialize the pipeline executor
-    job_buffer: Buffer = dicon.get("buffer")
-    pipeline = PipelineExecutor(
-        pipeline=[
-            # dicon.get("debug_step"),  # noqa: ERA001
-            dicon.get("job_repository_update_step"),
-            dicon.get("multi_manual_step"),
-            dicon.get("tranqu_step"),
-            dicon.get("estimator_step"),
-            dicon.get("ro_error_mitigation_step"),
-            job_buffer,
-            dicon.get("sse_step"),
-            dicon.get("device_gateway_step"),
-        ],
-        job_buffer=job_buffer,
-        exception_handler=exception_handler,
-    )
+    # Build the pipeline executor using the PipelineBuilder
+    pipeline = PipelineBuilder.build(gctx.config["pipeline_executor"], dicon)
 
     # Initialize the job fetcher
     job_fetcher: JobFetcher = dicon.get("job_fetcher")
diff --git a/core/src/oqtopus_engine_core/framework/__init__.py b/core/src/oqtopus_engine_core/framework/__init__.py
index 7f2f66a..9597a08 100644
--- a/core/src/oqtopus_engine_core/framework/__init__.py
+++ b/core/src/oqtopus_engine_core/framework/__init__.py
@@ -16,6 +16,7 @@
     TranspileResult,
 )
 from .pipeline import PipelineExecutor
+from .pipeline_builder import PipelineBuilder
 from .step import (
     DetachOnPostprocess,
     DetachOnPreprocess,
@@ -44,6 +45,7 @@
     "JoinOnPostprocess",
     "JoinOnPreprocess",
     "OperatorItem",
+    "PipelineBuilder",
     "PipelineExceptionHandler",
     "PipelineExecutor",
     "SamplingResult",
diff --git a/core/src/oqtopus_engine_core/framework/pipeline_builder.py b/core/src/oqtopus_engine_core/framework/pipeline_builder.py
new file mode 100644
index 0000000..cc28667
--- /dev/null
+++ b/core/src/oqtopus_engine_core/framework/pipeline_builder.py
@@ -0,0 +1,75 @@
+from typing import TYPE_CHECKING, Any
+
+from oqtopus_engine_core.framework import (
+    Buffer,
+    PipelineExceptionHandler,
+    PipelineExecutor,
+)
+from oqtopus_engine_core.utils.di_container import DiContainer
+
+if TYPE_CHECKING:
+    from oqtopus_engine_core.framework.step import Step
+
+
+class PipelineBuilder:
+    """PipelineBuilder: Construct a PipelineExecutor from config + DI Container.
+
+    This builder focuses *only* on building the PipelineExecutor.
+    It does NOT wire job fetchers, device fetchers, or repositories.
+    Those belong to the application initialization layer (app.py).
+    """
+
+    @staticmethod
+    def build(
+        pipeline_config: dict[str, Any],
+        dicon: DiContainer,
+    ) -> PipelineExecutor:
+        """Build a PipelineExecutor instance from the given pipeline configuration.
+
+        Expected config format (under the `pipeline_executor` root key):
+
+            pipeline_executor:
+              pipeline:
+                - job_repository_update_step
+                - multi_manual_step
+                - tranqu_step
+                - estimator_step
+                - ro_error_mitigation_step
+                - buffer
+                - sse_step
+                - device_gateway_step
+              job_buffer: buffer
+              exception_handler: exception_handler
+
+        Args:
+            pipeline_config : Dict[str, Any]
+                The configuration dictionary containing:
+                - "pipeline": list of component names in order
+                - "job_buffer": name of the buffer component
+                - "exception_handler": name of the exception handler component
+            dicon : DiContainer
+                The dependency injection container used to resolve component instances.
+
+        Returns:
+            PipelineExecutor: A fully constructed PipelineExecutor instance.
+
+        """
+        # pipeline nodes (steps and buffers)
+        pipeline: list[Buffer | Step] = [
+            dicon.get(name) for name in pipeline_config["pipeline"]
+        ]
+
+        # job buffer
+        job_buffer: Buffer = dicon.get(pipeline_config["job_buffer"])
+
+        # exception handler
+        exception_handler: PipelineExceptionHandler = dicon.get(
+            pipeline_config["exception_handler"]
+        )
+
+        # Construct PipelineExecutor
+        return PipelineExecutor(
+            pipeline=pipeline,
+            job_buffer=job_buffer,
+            exception_handler=exception_handler,
+        )
diff --git a/core/tests/oqtopus_engine_core/framework/test_pipeline_builder.py b/core/tests/oqtopus_engine_core/framework/test_pipeline_builder.py
new file mode 100644
index 0000000..b8905d8
--- /dev/null
+++ b/core/tests/oqtopus_engine_core/framework/test_pipeline_builder.py
@@ -0,0 +1,116 @@
+import pytest
+
+from oqtopus_engine_core.framework import PipelineBuilder, PipelineExecutor
+from oqtopus_engine_core.utils.di_container import DiContainer
+
+# -----------------------
+# Fake classes for testing
+# -----------------------
+
+class FakeStep:
+    """A minimal fake Step used for testing PipelineBuilder."""
+    pass
+
+
+class FakeBuffer:
+    """A minimal fake Buffer used for testing PipelineBuilder."""
+    pass
+
+
+class FakeExceptionHandler:
+    """A minimal fake exception handler for testing."""
+    pass
+
+
+# --------------------------------
+# Simple Fake DI Container for testing
+# --------------------------------
+
+class FakeDiContainer(DiContainer):
+    """
+    A simple mock of DiContainer that stores objects in a registry
+    and returns them when get(name) is called.
+    """
+
+    def __init__(self, registry):
+        self._registry = registry
+
+    def get(self, name):
+        return self._registry[name]
+
+
+# --------------------------------
+# Test Cases
+# --------------------------------
+
+def test_pipeline_builder_builds_executor_correctly():
+    """
+    Test that PipelineBuilder.build constructs a PipelineExecutor with:
+
+      - The correct ordered list of pipeline components
+      - The correct job buffer instance
+      - The correct exception handler instance
+
+    This ensures that PipelineBuilder performs DI lookups correctly
+    and passes the resolved objects to PipelineExecutor.
+    """
+
+    # Arrange: create fake components
+    step1 = FakeStep()
+    step2 = FakeStep()
+    buffer = FakeBuffer()
+    exception_handler = FakeExceptionHandler()
+
+    # Fake DI registry
+    registry = {
+        "step1": step1,
+        "step2": step2,
+        "buffer": buffer,
+        "exception_handler": exception_handler,
+    }
+
+    dicon = FakeDiContainer(registry)
+
+    # Config following the expected format
+    pipeline_config = {
+        "pipeline": ["step1", "step2", "buffer"],
+        "job_buffer": "buffer",
+        "exception_handler": "exception_handler",
+    }
+
+    # Act
+    executor = PipelineBuilder.build(pipeline_config, dicon)
+
+    # Assert
+    assert isinstance(executor, PipelineExecutor)
+    assert executor._pipeline == [step1, step2, buffer]
+    assert executor._job_buffer is buffer
+    assert executor._exception_handler is exception_handler
+
+
+def test_pipeline_builder_raises_key_error_if_missing_component():
+    """
+    Test that PipelineBuilder.build raises a KeyError if any component
+    listed in the config is missing from the DI container.
+
+    This confirms that the builder does not silently ignore unresolved
+    components and fails fast when configuration is incorrect.
+    """
+
+    # Fake DI without all necessary components
+    registry = {
+        "step1": FakeStep(),
+        # step2 is missing
+        "buffer": FakeBuffer(),
+        "exception_handler": FakeExceptionHandler(),
+    }
+    dicon = FakeDiContainer(registry)
+
+    pipeline_config = {
+        "pipeline": ["step1", "step2"],   # step2 will cause KeyError
+        "job_buffer": "buffer",
+        "exception_handler": "exception_handler",
+    }
+
+    with pytest.raises(KeyError):
+        PipelineBuilder.build(pipeline_config, dicon)
diff --git a/docs/design/pipeline_execution.md b/docs/design/pipeline_execution.md
index fee376f..7413864 100644
--- a/docs/design/pipeline_execution.md
+++ b/docs/design/pipeline_execution.md
@@ -22,7 +22,73 @@ The pipeline execution model is based on three key principles:
 
 The PipelineExecutor orchestrates all of these behaviors.
 
-## 2. Pipeline Structure
+## 2. Pipeline Construction from `config.yaml`
+
+## 2.1 Defining the Pipeline in `config.yaml`
+
+OQTOPUS Engine Core supports **configuration-driven pipeline construction**.
+A pipeline does not need to be hard-coded in Python—the engine can create a
+`PipelineExecutor` directly from the YAML configuration.
+
+The `pipeline_executor` section defines:
+
+- the ordered list of pipeline elements,
+- which component acts as the job buffer,
+- which component handles exceptions during pipeline execution.
+
+A corresponding entry in `di_container.registry` supplies the concrete
+implementations for each name.
+
+Example:
+
+```yaml
+pipeline_executor:
+  pipeline:
+    - job_repository_update_step
+    - multi_manual_step
+    - tranqu_step
+    - estimator_step
+    - ro_error_mitigation_step
+    - buffer
+    - sse_step
+    - device_gateway_step
+  job_buffer: buffer
+  exception_handler: pipeline_exception_handler
+
+di_container:
+  registry:
+    job_repository_update_step:
+      _target_: oqtopus_engine_core.steps.JobRepositoryUpdateStep
+
+    ...
+```
+
+### 2.2 How the Engine Uses This Configuration
+
+The engine reads `pipeline_executor` and retrieves all components from the dependency-injection container:
+
+- every pipeline element (steps and buffers) is retrieved from the DI registry;
+- the element order in YAML becomes the actual traversal order;
+- the job buffer is assigned to `job_buffer`;
+- the exception handler is assigned to `exception_handler`.
+
+Internally, the engine uses `PipelineBuilder.build()` to construct the executor:
+
+- read the ordered list under `pipeline`;
+- retrieve each component via `dicon.get(name)`;
+- retrieve `job_buffer` and `exception_handler`;
+- create a `PipelineExecutor` with these objects.
+
+This allows:
+
+- fully declarative pipeline definition,
+- environment-specific override via YAML or environment variables,
+- consistent dependency management across all components,
+- a clean separation between "pipeline construction" and "pipeline execution."
+
+The following sections describe **how the constructed pipeline executes jobs** including two-phase traversal, buffers, workers, split/join semantics, detach, and error handling.
+
+## 3. Pipeline Structure
 
 A pipeline consists of an ordered sequence of **elements**:
 
@@ -37,11 +103,11 @@ Step A → Step B → Buffer X → Step C → Step D
 
 Jobs advance through these elements depending on the current phase of execution.
 
-## 3. Two-Phase Traversal
+## 4. Two-Phase Traversal
 
 The executor performs job traversal in two distinct phases:
 
-### 3.1 pre-process Phase (Forward Traversal)
+### 4.1 pre-process Phase (Forward Traversal)
 
 - Traversal starts at index 0.
 - For each step, the executor calls `pre_process(job, jctx)`.
@@ -56,7 +122,7 @@ This phase is responsible for:
 - performing transformations or expansions (e.g., splitting),
 - interacting with external services before execution.
 
-### 3.2 post-process Phase (Backward Traversal)
+### 4.2 post-process Phase (Backward Traversal)
 
 - Traversal starts at the end of the pipeline.
 - For each step, the executor calls `post_process(job, jctx)`.
@@ -72,18 +138,18 @@ The backward phase is typically used for:
 - applying final transforms,
 - merging metadata into the job.
 
-## 4. Buffers and Worker Tasks
+## 5. Buffers and Worker Tasks
 
 Buffers create asynchronous execution boundaries.
 
-### 4.1 Enqueuing and Halting
+### 5.1 Enqueuing and Halting
 
 During pre-process traversal:
 
 - when a job hits a buffer, the executor posts it into the buffer queue;
 - the executor returns control immediately without traversing downstream elements.
 
-### 4.2 Worker Execution
+### 5.2 Worker Execution
 
 Each buffer has a dedicated worker, started when the pipeline begins.  
 A worker performs:
@@ -98,14 +164,14 @@ This design allows:
 - natural backpressure (queue length control),
 - asynchronous decoupling between pipeline segments.
 
-### 4.3 Buffers in post-process Phase
+### 5.3 Buffers in post-process Phase
 
 Buffers are ignored during backward traversal.  
 Jobs do **not** stop or queue in the post-process phase.
 
-## 5. Split Execution
+## 6. Split Execution
 
-### 5.1 Purpose of Split Steps
+### 6.1 Purpose of Split Steps
 
 A split step divides one job into multiple independent child jobs.  
 This enables functionalities such as:
@@ -115,7 +181,7 @@ This enables functionalities such as:
 - job replication for sampling or batching,
 - custom branching logic.
 
-### 5.2 How Splitting Works
+### 6.2 How Splitting Works
 
 Depending on the type of step:
 
@@ -137,7 +203,7 @@ The executor automatically manages:
 - tracking of remaining children,
 - dispatching each child into the pipeline.
 
-### 5.3 Child Job Independence
+### 6.3 Child Job Independence
 
 Each child job:
 
@@ -149,9 +215,9 @@ Each child job:
 
 This creates a full job tree.
 
-## 6. Join Execution
+## 7. Join Execution
 
-### 6.1 Purpose of Join Steps
+### 7.1 Purpose of Join Steps
 
 Join steps re-synchronize parallel execution paths.  
 They aggregate child results to resume the parent job.
@@ -163,7 +229,7 @@ Typical use cases:
 - collecting metadata or logs,
 - post-processing on aggregated outputs.
 
-### 6.2 How Join Works
+### 7.2 How Join Works
 
 A join step is executed when the **last** living child job reaches it.
 
@@ -188,7 +254,7 @@ This ensures:
 - no duplicate join invocations,
 - safe synchronization before parent continues.
 
-### 6.3 Join During Forward vs Backward Traversal
+### 7.3 Join During Forward vs Backward Traversal
 
 Join steps may live in the pipeline during pre-process or post-process phases:
 
@@ -197,7 +263,7 @@ Join steps may live in the pipeline during pre-process or post-process phases:
 
 This enables expressive control-flow patterns.
 
-## 7. Composite Job Trees and Structured Concurrency
+## 8. Composite Job Trees and Structured Concurrency
 
 Split/Join semantics create a structured execution pattern analogous to classical fork/join models:
 
@@ -214,7 +280,7 @@ This ensures:
 - clear control of job lifecycle,
 - safe aggregation points.
 
-## 8. Detach Execution
+## 9. Detach Execution
 
 Detach steps allow part of a job’s pipeline traversal to continue in a
 separate coroutine.  
@@ -249,7 +315,7 @@ Detaching preserves:
 This mechanism is useful for offloading long-running pipeline segments
 without requiring additional worker threads.
 
-## 9. Exception Handling
+## 10. Exception Handling
 
 Each step is executed in a “safe call” wrapper:
 
@@ -260,14 +326,14 @@ Each step is executed in a “safe call” wrapper:
 
 This prevents inconsistent pipeline states and provides clear diagnostics.
 
-## 10. Step History Tracking
+## 11. Step History Tracking
 
 Each job maintains an execution history stored inside its `JobContext`
 under the field `step_history`.  
 This history records **which step was executed**, **in which phase**, and
 **at which pipeline index** during traversal.
 
-### 10.1 Representation
+### 11.1 Representation
 
 `step_history` is a **list of tuples** of the form: `(phase, cursor)`
 
@@ -291,7 +357,7 @@ jctx.step_history == [
 
 This structure provides a precise trace of the job’s movement through the pipeline.
 
-### 10.2 Parent and Child Jobs
+### 11.2 Parent and Child Jobs
 
 When a split step creates child jobs:
 
@@ -329,7 +395,7 @@ Final accumulated histories:
 - **Children**:  
   `[("pre-process", 2), ("post-process", 2), ("post-process", 1)]`
 
-## 11. Summary
+## 12. Summary
 
 The pipeline execution model supports:
 


Reviews:

Review comments:

PR comments: