Skill: exenvitor/k8s-prefect-flow-builder (version 1.0.0)

--- name: prefect-flow-builder description: Build, modify, and review Prefect-based offline orchestration in this repository. Use when adding a new Prefect flow, wrapping an existing offline computation as a flow or task, editing prefect.yaml, splitting deployments by resource profile, configuring resources or concurrency, or aligning CI, Prefect, and Kubernetes deployment behavior. --- # Prefect Flow Builder ## Overview Use this skill to add or refactor Prefect-managed offline workflows in this repository without mixing orchestration concerns into business logic. ## Workflow 1. Classify the change. - New flow or major refactor: read `references/flow-design.md` and `references/template-prefect-yaml.md`. - Deployment-only or config change: read `references/deployment-patterns.md`. - Resource or concurrency tuning: read `references/resources-and-concurrency.md`. - If the system model is unclear, read `references/architecture.md` first. 2. Keep orchestration separate from compute. - Put heavy business logic in reusable job or service modules under `src/core/...`. - Keep Prefect wrappers in `src/prefect_flows/...`. - Use tasks for independently observable units or meaningful side effects; avoid exploding one logical step into many tiny tasks just for structure. - Choose task invocation mode deliberately: direct call for simple serial execution, `.submit()` or `.map()` for in-flow concurrency, and `.delay()` only for background execution on separate infrastructure. - If a child flow is intentionally serial, call tasks directly inside the loop and let the child flow own error aggregation and final failure semantics. Do not default to `.submit()` just because the unit is a task. - Reserve `.submit()` for cases that actually need Prefect futures, parallel fan-out, or non-blocking wait and collection behavior, and make sure terminal futures are resolved before the flow returns. - Introduce child flows only when you need a separate scaling, resource, or failure boundary. 3. Put concurrency controls at the right layer. - Use task-runner concurrency for in-flow task execution only; it is not a substitute for deployment or infrastructure throttling. - Use deployment, work-pool, worker, or work-queue limits to control how many flow runs the platform launches. - Use tag-based concurrency limits when many tasks across flows share an external bottleneck such as a database, API, or memory-heavy resource. 4. Treat `prefect.yaml` as deployment source of truth. - Put deployment names, schedules, work pool selection, resources, concurrency policies, and deployment parameter defaults in `prefect.yaml`. - Let CI provide only deploy-time values such as `PREFECT_API_URL` and `PREFECT_DEPLOY_IMAGE`. - Keep runtime business env in Kubernetes via `env_from`, not CI. - Keep infrastructure choices aligned with Prefect's worker model: deployments target work pools, and workers poll compatible pools to execute runs. 5. Make failures observable. - Raise exceptions instead of returning non-zero codes silently. - Include key context in exception messages and logs. - Use readable task and flow names so Prefect UI can identify the failing unit quickly. - When a batch must continue after per-item failures, convert each item into a structured result, finish the batch, then raise once at the flow boundary if the aggregate outcome should be failed. 6. Validate in this order. - Run focused tests for changed flow and job code. - Review `prefect.yaml` for deployment or resource drift. - Trigger a small manual run before enabling or changing schedules. - If the question is about Prefect semantics rather than repo conventions, verify against the official v3 docs before codifying a pattern. ## Repository Anchors - Flow entrypoints: `src/prefect_flows/` - Reusable compute logic: `src/core/` - Deployment definitions: `prefect.yaml` - Team guide: `docs/cg_offline_prediction/prefect_orchestration_overview.md` ## Prefect v3 Defaults - Tasks support three distinct execution modes in v3: direct call blocks and returns a result, `.submit()` returns a `PrefectFuture` for concurrent execution in the same flow, and `.delay()` is for background execution on separate workers. - Task runners are optional. If you are not intentionally using concurrency, do not introduce `.submit()` or a task runner configuration just to preserve the task decorator. - Task states are orchestrated client-side and may appear with eventual consistency in the UI, so design recovery and resume logic around durable business outcomes instead of assuming every intermediate task-state transition is the source of truth. ## References - `references/architecture.md` for the system model and lifecycle. - `references/deployment-patterns.md` for deployment design and config ownership. - `references/flow-design.md` for wrapping existing jobs as flows and tasks. - `references/resources-and-concurrency.md` for sizing and concurrency decisions. - `references/template-prefect-yaml.md` for reusable deployment templates. - Prefect v3 Tasks: https://docs.prefect.io/v3/concepts/tasks - Prefect v3 Task runners: https://docs.prefect.io/v3/concepts/task-runners/ - Prefect v3 Flows: https://docs.prefect.io/v3/concepts/flows - Prefect v3 Deployments: https://docs.prefect.io/v3/concepts/deployments - Prefect v3 Workers: https://docs.prefect.io/v3/concepts/workers - Prefect v3 Tag-based concurrency limits: https://docs.prefect.io/v3/concepts/tag-based-concurrency-limits

path: references/architecture.md
content: # Architecture

## Core relationship

- Code defines flows, tasks, reusable job logic, and `prefect.yaml`.
- CI builds and pushes the image.
- CI runs `prefect deploy --prefect-file prefect.yaml --all`.
- Prefect stores deployment definitions and creates flow runs.
- A Kubernetes work pool starts pods to execute those runs.
- Runtime env comes from Kubernetes `Secret` or `ConfigMap`, not from CI.

## Deployment lifecycle

1. Implement or update flow and deployment code.
2. CI builds and pushes an image.
3. CI exports `PREFECT_API_URL` and `PREFECT_DEPLOY_IMAGE`.
4. CI runs `prefect deploy`.
5. Prefect updates deployments.
6. A schedule or operator triggers a run later.

`prefect deploy` registers deployments. It does not execute business work.

## Runtime execution model

1. A deployment is triggered by schedule or manually.
2. Prefect creates a flow run.
3. The work pool starts a Kubernetes pod for the run.
4. The pod receives runtime env through `env_from`.
5. The flow runs tasks directly, or dispatches child flows if fan-out is needed.

## Configuration ownership

- Code defaults: light runtime defaults such as retry counts.
- `prefect.yaml`: deployment behavior and operational defaults.
- CI: deploy-time values only.
- Kubernetes: runtime business configuration and secrets.


path: references/deployment-patterns.md
content: # Deployment Patterns

## Single deployment

Use one deployment when:
- scheduled and manual runs share a similar resource profile;
- one operational identity is enough;
- there is no need to split scaling or failure boundaries.

## Separate scheduled and manual deployments

Split deployments when:
- scheduled runs are lightweight but manual or historical runs are heavier;
- schedules and manual triggers need different defaults;
- concurrency or resource policy should differ.

## Parent and child deployments

Use a parent flow plus child flow when:
- one orchestration run must fan out into multiple independent batches;
- child work needs its own pod resources;
- child failures should be isolated and visible.

Keep deployment names in `prefect.yaml`. Use deployment names for operational identity and flow names for logical identity.

## Work pool and runtime env

In this repository, deployments should rely on:
- `work_pool.name` in `prefect.yaml`;
- `job_variables.image: "{{ $PREFECT_DEPLOY_IMAGE }}"`;
- `job_variables.env_from` pointing to Kubernetes runtime config.

The work pool `base_job_template` must support `env_from` and map it to Kubernetes `envFrom`.

## Concurrency policies

Use deployment-level `concurrency_limit` when one deployment must serialize itself.
Typical pattern:

```yaml
concurrency_limit:
  limit: 1
  collision_strategy: CANCEL_NEW
```

This applies to one deployment only. Do not treat it as a cross-deployment global lock.


path: references/flow-design.md
content: # Flow Design

## Start from existing compute logic

Before adding Prefect wrappers:
- extract heavy business logic into a reusable module under `src/core/...`;
- make it callable without Prefect;
- define explicit inputs and a structured result.

## Choose the orchestration boundary

- Use a flow for orchestration.
- Use a task for one independently observable execution unit.
- Use a child flow only when you need a separate scaling, resource, or failure boundary.

## Failure propagation

- Do not return non-zero values silently.
- Raise exceptions so Prefect marks the task or flow as failed.
- Add key context to exception messages so Prefect state messages remain useful after pod logs expire.

## Naming and logs

Use readable names in Prefect UI:
- `task_run_name` for tasks;
- `flow_run_name` for child flows launched by a parent.

Prefer names that expose the unit being processed, such as a batch id or target id.

Keep logs short and structured. A failed unit should emit one log line with enough context to debug without re-reading the pod log.

## Validation checklist

Before enabling a schedule:
- run focused tests;
- verify failure propagation;
- verify task and flow names in Prefect UI;
- run one small manual validation.


path: references/resources-and-concurrency.md
content: # Resources and Concurrency

## Requests vs limits

- `requests` are the resources Kubernetes tries to reserve for the pod.
- `limits` are the maximum resources the container may consume.
- Exceeding `limits.memory` can OOM-kill the pod.
- Exceeding `limits.cpu` throttles the container.

## Starting pattern

- Parent or orchestration flows usually need low CPU and memory.
- Compute-heavy child flows should be sized by the largest in-memory working set and expected CPU pressure.

## Tuning loop

1. Start with a small manual run.
2. Observe memory peak, CPU usage, task duration, retries, and OOM events.
3. Tune these together:
   - batch size;
   - concurrency;
   - chunk size or per-task processing window;
   - pod requests and limits.

Do not tune resources in isolation. Throughput and memory footprint are usually coupled.

## When to split deployments

Split deployments instead of overloading one profile when:
- scheduled runs are much lighter than manual or historical runs;
- parent and child work have different resource needs;
- one execution mode needs more concurrency but another needs more memory.


path: references/template-prefect-yaml.md
content: # Template `prefect.yaml`

Use this as a starting point and rename fields for the job you are introducing.

```yaml
name: your-project
prefect-version: 3.6.21

pull:
- prefect.deployments.steps.set_working_directory:
    directory: /app

deployments:
- name: your-job-scheduled
  tags: ["offline", "scheduled"]
  description: "Scheduled offline job."
  concurrency_limit:
    limit: 1
    collision_strategy: CANCEL_NEW
  schedule:
    cron: "0 * * * *"
    timezone: "UTC"
    active: false
  flow_name: your_flow_name
  entrypoint: src/prefect_flows/your_flow.py:your_flow
  work_pool:
    name: k8s-pool
    job_variables:
      image: "{{ $PREFECT_DEPLOY_IMAGE }}"
      env_from:
        - secretRef:
            name: your-runtime-secret
      resources:
        requests:
          cpu: "100m"
          memory: "512Mi"
        limits:
          cpu: "1"
          memory: "1Gi"
  parameters:
    execution_mode: scheduled
    run_at: null

- name: your-job-manual
  tags: ["offline", "manual"]
  description: "Manual run template."
  schedule: {}
  flow_name: your_flow_name
  entrypoint: src/prefect_flows/your_flow.py:your_flow
  work_pool:
    name: k8s-pool
    job_variables:
      image: "{{ $PREFECT_DEPLOY_IMAGE }}"
      env_from:
        - secretRef:
            name: your-runtime-secret
      resources:
        requests:
          cpu: "100m"
          memory: "1Gi"
        limits:
          cpu: "2"
          memory: "2Gi"
  parameters:
    execution_mode: manual
    run_at: null
    start_at: null
```

## Template usage notes

- Keep deployment behavior in `prefect.yaml`, not in CI shell logic.
- Keep runtime business env in Kubernetes via `env_from`.
- Use separate deployments when execution modes need different defaults or resources.
- If a flow fans out into heavy compute batches, consider a dedicated child deployment.


path: agents/openai.yaml
content: interface:
  display_name: "Prefect Flow Builder"
  short_description: "Build and modify Prefect flow setups"
  default_prompt: "Use $prefect-flow-builder to add or refactor a Prefect-based offline workflow."


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)