Skill: quochungto/bookforge-batch-pipeline-designer (version 1.0.0)

--- name: batch-pipeline-designer description: | Design batch data processing pipelines for large-scale, bounded datasets processed offline. Use when building ETL workflows, processing logs or clickstream data at scale, generating ML feature pipelines or search indexes, or joining two large datasets that cannot fit in memory. Trigger phrases: "design a batch pipeline", "should I use Spark or MapReduce", "how do I join two large datasets", "build an ETL workflow", "process server logs at scale", "how do I handle skewed data in joins", "implement PageRank on a distributed graph", "design an offline processing job". Covers MapReduce vs dataflow engines (Spark, Flink, Tez), three join strategies (sort-merge, broadcast hash, partitioned hash) with selection criteria, graph processing via the Pregel/BSP model, and fault tolerance via materialization vs recomputation. Does not apply to unbounded input streams (see stream-processing-designer) or low-latency OLTP query serving. Produces a pipeline architecture recommendation with engine choice, join strategy, and fault tolerance approach. model: sonnet context: 1M execution: tier: 2 mode: hybrid inputs: - type: document description: "System requirements, data flow diagrams, schema definitions, or architecture descriptions" - type: codebase description: "Existing pipeline code or infrastructure configs to understand current setup" - type: none description: "Skill can work from a verbal description of the processing requirements" tools-required: [Read, TodoWrite] tools-optional: [Grep, Bash] environment: "Can run from any directory; codebase access enables analysis of existing pipeline code" depends-on: - oltp-olap-workload-classifier --- # Batch Pipeline Designer ## When to Use Use this skill when you are designing or evaluating a **batch processing pipeline** — a system that reads a bounded, fixed-size dataset, runs a job over it, and produces output. Batch jobs are scheduled periodically (hourly, daily), tolerate high latency, and are measured by throughput rather than response time. This skill applies when: - Processing large datasets offline (logs, clickstream, database snapshots, file dumps) - Building ETL workflows to move data between systems - Generating derived datasets: search indexes, ML training features, recommendation model inputs, aggregated reports - Implementing multi-step data transformations across distributed storage - Joining two or more large datasets that cannot fit in memory on a single machine - Processing graph-structured data (social graphs, link graphs, dependency graphs) in bulk **This skill does NOT apply to:** - Stream processing where input is unbounded (see `stream-processing-designer`) - Online query serving where latency < 1 second is required - Transactional workloads (see `oltp-olap-workload-classifier` to confirm the workload type) **Dependency check:** If the workload classification is uncertain, invoke `oltp-olap-workload-classifier` first to confirm this is a batch workload before proceeding. ## Context & Input Gathering ### Input Sufficiency Check ``` User prompt → Extract: data sources, transformations required, output destination ↓ Environment → Check: existing pipeline code, schema files, infrastructure configs ↓ Gap analysis → Do I know the input volume, join requirements, latency tolerance? ↓ Missing critical info? ──YES──→ ASK (one question at a time) │ NO ↓ PROCEED with pipeline design ``` ### Required Context (must have — ask if missing) - **Input data characteristics:** What datasets are being processed? What are their approximate sizes? → Check prompt for: dataset names, "gigabytes", "terabytes", "millions of records", log file descriptions → Check environment for: schema files, data flow docs, existing pipeline configs → If still missing, ask: "What are the input datasets, and roughly how large is each one? (e.g., 100 GB user event log + 10 GB user profile database)" - **Required transformations and joins:** What operations need to happen — filtering, aggregation, joins between datasets? → Check prompt for: SQL-like language ("group by", "join on", "aggregate"), descriptions of data enrichment or correlation → If still missing, ask: "What does the pipeline need to produce? For example: 'count page views per URL', 'join user events with user profiles', or 'build a feature matrix for ML training'." - **Output destination:** Where does the processed data go — a database, search index, file system, another pipeline? → Check prompt for: destination systems, "load into", "write to", "serve via" → If still missing, ask: "Where does the output of this pipeline need to go, and how will it be consumed? (e.g., loaded into Elasticsearch, written to HDFS for downstream jobs, served via a key-value store)" - **Latency tolerance:** Is this a daily job, hourly, or near-real-time? How long can the job take to complete? → Check prompt for: "daily", "hourly", "SLA", "must finish by", scheduling language → If still missing, ask: "What is the acceptable job completion time? For example: 'must finish within 6 hours of midnight' or 'runs weekly, no strict deadline'." ### Observable Context (gather from environment) - **Existing infrastructure:** Look for Hadoop, Spark, Flink, or Airflow configurations that constrain engine choice. → Look for: `pom.xml` with spark/hadoop deps, `requirements.txt` with pyspark, `airflow/dags/`, `Dockerfile` with cluster images → If unavailable: assume greenfield, no constraint on engine choice - **Data partitioning:** Check whether input datasets are already partitioned and by what key. → Look for: HDFS directory structures, partition specs in Hive metastore configs, dataset metadata files → If unavailable: assume unpartitioned, sort-merge join is safe default - **Skew indicators:** Check whether any join keys are known to have high-cardinality hot keys (e.g., celebrity users in a social graph, viral content IDs). → Look for: comments about hot keys, prior skew handling code, "top user" or "trending" concepts in data model → If unavailable: assume uniform distribution until data profile reveals otherwise ### Default Assumptions - If engine is unspecified and infrastructure allows it: recommend a dataflow engine (Spark or Flink) over raw MapReduce — they execute significantly faster for most workloads. - If join dataset sizes are unspecified: default to sort-merge join (safest for unknown sizes, no memory assumptions). - If output destination is unspecified: write to distributed filesystem (HDFS, S3) — composable, immutable, safe. - If fault tolerance requirement is unspecified: use materialization for jobs longer than 30 minutes; use in-memory recomputation for shorter jobs. ### Questioning Guidelines Ask ONE question at a time, most critical first (input characteristics before transformation details before output). Show what you already know from the environment before asking. State WHY you need the information. ## Process Use `TodoWrite` to track steps before beginning. ``` TodoWrite([ { id: "1", content: "Confirm workload is batch; gather input/output characteristics", status: "pending" }, { id: "2", content: "Select processing engine: MapReduce vs dataflow engine", status: "pending" }, { id: "3", content: "Design join strategy for any cross-dataset operations", status: "pending" }, { id: "4", content: "Design multi-step workflow and intermediate state handling", status: "pending" }, { id: "5", content: "Assess graph processing needs (if applicable)", status: "pending" }, { id: "6", content: "Design fault tolerance strategy", status: "pending" }, { id: "7", content: "Specify output destination and loading strategy", status: "pending" }, { id: "8", content: "Produce pipeline architecture document", status: "pending" } ]) ``` --- ### Step 1: Confirm Workload Classification **ACTION:** Verify the workload is genuinely batch before proceeding. Determine input volume, update frequency, and latency tolerance. **WHY:** The boundary between batch and stream processing determines which design space applies. A pipeline that processes data "every hour" sounds like batch — but if the latency requirement is under 5 minutes, it is stream processing in disguise. Getting this wrong leads to building the wrong system. The key distinguishing property of batch is that the input is **bounded** — it has a known, fixed size at the time the job starts. A job knows when it has finished reading all input, so it can eventually complete. **Classification signals:** - **Batch:** Input is a file dump, database snapshot, or accumulated log from a fixed time window. Job runs periodically. Latency measured in minutes to hours. Primary metric is throughput. - **Stream:** Input arrives continuously and is never "done." Latency measured in milliseconds to seconds. See `stream-processing-designer` instead. **IF** workload classification is ambiguous → invoke `oltp-olap-workload-classifier` first **IF** clearly batch → proceed to Step 2 Mark Step 1 complete in TodoWrite. --- ### Step 2: Select the Processing Engine **ACTION:** Choose between MapReduce and a dataflow engine (Spark, Flink, or Tez) for this pipeline. **WHY:** MapReduce is simple and robust — it fully materializes intermediate state to disk, tolerates frequent task failures without any data loss, and runs safely on preemptible, low-priority compute. But this robustness has a cost: every intermediate result is written to the distributed filesystem and read back by the next stage. For complex pipelines with many stages, this disk I/O dominates runtime. Dataflow engines (Spark, Flink, Tez) treat the entire workflow as one job, pass data between operators in memory or local disk rather than the distributed filesystem, and skip sorting where it is not required. They execute the same computation significantly faster — often orders of magnitude for multi-stage pipelines — while providing equivalent fault tolerance via lineage tracking (Spark's Resilient Distributed Dataset) or operator checkpointing (Flink). **Decision framework:** | Factor | Prefer MapReduce | Prefer Dataflow Engine (Spark/Flink/Tez) | |--------|-----------------|------------------------------------------| | Pipeline stages | 1-3 stages | 4+ stages (many intermediate materializations in MR) | | Task preemption risk | High (shared cluster, MR runs low-priority) | Low to moderate (dedicated cluster) | | Existing infrastructure | Hadoop-only environment | Any modern cluster | | Fault tolerance model | Must tolerate very frequent task kills | Standard hardware failure tolerance | | Processing model | Batch only | Batch + can evolve toward streaming | | Join complexity | Simple reduce-side joins | Complex multi-way joins, broadcast joins | **Engine-specific selection within dataflow engines:** - **Spark:** Batch processing, machine learning (MLlib), iterative algorithms. Rich ecosystem. Best default choice. - **Flink:** Lowest latency for near-real-time batch; strong streaming integration; best when pipeline must eventually handle stream input. - **Tez:** Thin optimization layer over YARN; lowest overhead when Pig/Hive workflows already exist and just need faster execution. **Note on high-level languages:** In most cases, you do not write raw MapReduce or dataflow operator code. Higher-level tools (Hive, Spark SQL, Pig, Cascading, Crunch) compile to the underlying engine and include query optimizers that automatically select the best join algorithm. Declare joins relationally and let the optimizer decide — this is preferred unless fine-grained control is needed. Mark Step 2 complete in TodoWrite. --- ### Step 3: Design the Join Strategy **ACTION:** For any operation that correlates records from two or more datasets, select and specify the appropriate join algorithm from the three available strategies. **WHY:** Joins are the most expensive and failure-prone operation in batch pipelines. The naive approach — querying a remote database for every record — is orders of magnitude slower than the pipeline's normal throughput, and it makes the job nondeterministic (the remote database may change during the job). The correct approach is to bring all the data needed for a join to the same place as a local operation. Three strategies accomplish this, each suited to different data size and partitioning conditions. **Strategy 1 — Sort-Merge Join (reduce-side join)** Use when: both datasets are large, sizes are unknown or similar, or datasets have no pre-existing partitioning guarantees. How it works: Both datasets go through a mapper that emits records keyed by the join key. The shuffle/sort phase collects all records with the same key to the same reducer partition. The reducer sees all records for a given key together and performs the join. A secondary sort ensures that records from one dataset (e.g., a "profile" record) arrive before records from the other dataset (e.g., "activity events"), so the reducer can hold the profile record in memory while iterating through activity events without buffering everything. Cost: All data is shuffled over the network and sorted. Expensive for large datasets but correct and scalable with no assumptions about data layout. **Strategy 2 — Broadcast Hash Join (map-side join, small table)** Use when: one dataset is small enough to fit entirely in memory on each mapper (typically < a few GB). How it works: Each mapper loads the small dataset into an in-memory hash table at startup (the small dataset is "broadcast" to all mappers). The mapper then scans the large dataset record by record, looking up each record's join key in the hash table. No reducer is needed; no shuffle occurs. This is the fastest possible join when the small table fits in memory. Cost: Requires the small table to fit in memory. The small table is replicated to every mapper — memory pressure scales with the number of map tasks running concurrently. **Strategy 3 — Partitioned Hash Join (map-side join, co-partitioned tables)** Use when: both datasets are large BUT are already partitioned by the same key, using the same hash function, into the same number of partitions (e.g., both output from prior MapReduce jobs with the same partitioning scheme). How it works: Because both datasets are partitioned identically, mapper N for the large dataset only needs data from partition N of the small dataset. Each mapper loads one partition of the smaller table into memory and scans the corresponding partition of the larger table. No shuffle or reducer needed. Cost: Requires both datasets to be co-partitioned — a strong prerequisite. This metadata (partitioning key, function, partition count) must be known and consistent between datasets. **Handling skew in joins:** If a join key has extreme hot spots (a "celebrity" user ID with millions of events, a viral content ID), a standard sort-merge join will send all records with that key to a single reducer, causing a straggler that blocks the entire pipeline. Mitigation options: - **Skewed join (Pig):** Run a sampling job first to identify hot keys. Route hot key records to multiple reducers randomly; replicate the other dataset's records for that key to all those reducers. - **Sharded join (Crunch):** Specify hot keys explicitly; framework handles replication automatically. - **Two-stage aggregation:** For GROUP BY on hot keys, first reduce to a partial aggregate per reducer (reducing volume), then combine across reducers. Mark Step 3 complete in TodoWrite. --- ### Step 4: Design the Multi-Step Workflow **ACTION:** Decompose the full pipeline into stages, specify the data flow between stages, and decide how intermediate state is managed. **WHY:** A single MapReduce or dataflow job can only solve a limited range of problems. Complex pipelines require chaining multiple jobs — for example, a recommendation system may use 50-100 MapReduce jobs. The key architectural decision is how intermediate results are passed between stages: full materialization to the distributed filesystem (MapReduce style) vs. streaming through memory/local disk (dataflow engine style). This choice determines both performance and fault recovery behavior. **Workflow structure decisions:** 1. **Identify job dependencies:** Which jobs produce the input for which other jobs? Draw this as a directed acyclic graph (DAG). A job can only start once all its upstream dependencies have completed successfully — partial output from failed jobs is discarded. 2. **Intermediate state strategy:** - **Full materialization (MapReduce):** Each job writes output to a named directory in HDFS; the next job reads from that directory. The intermediate data is durable, replicated, and independently inspectable. Cost: extra I/O, extra network replication, straggler delays (must wait for all tasks in a stage before next stage starts). Benefit: any job can be restarted independently; output can be read by multiple downstream jobs; easy debugging (inspect intermediate files). - **Pipelining (Spark/Flink):** Intermediate state between operators is kept in memory or local disk. Operators start as soon as their upstream operator has produced some output — no waiting for the entire stage to complete. Cost: if a node fails, the operator's output must be recomputed from the last checkpoint or original input. Benefit: much lower latency for multi-stage pipelines. 3. **Workflow scheduler:** For MapReduce-based pipelines with many jobs, use a workflow scheduler (Apache Airflow, Luigi, Azkaban, Oozie) to manage job dependencies, retries, backfills, and monitoring. Dataflow engines handle workflow dependencies internally. 4. **Unix philosophy applied:** Design each pipeline stage to do one thing well, read from a well-defined input format, and write to a well-defined output format. Inputs are treated as immutable — a stage never modifies its input. This means any stage can be rerun safely, old outputs can be kept in parallel directories while new outputs are being computed, and rollbacks are possible by switching back to a previous output directory. Mark Step 4 complete in TodoWrite. --- ### Step 5: Assess Graph Processing Needs (if applicable) **ACTION:** If the pipeline operates on graph-structured data (social graphs, web link graphs, dependency graphs), determine whether the Pregel/bulk synchronous parallel model is appropriate. **WHY:** Many graph algorithms are iterative — they traverse edges, propagate information, and repeat until convergence (PageRank, shortest path, connected components, transitive closure). Implementing these in plain MapReduce is inefficient because MapReduce reads the entire input dataset and writes the entire output dataset on every iteration, even if only a small part of the graph changed. The Pregel model (also called bulk synchronous parallel, or BSP) addresses this by keeping vertex state in memory across iterations and only processing vertices that have received new messages. **How Pregel works:** - The graph is stored in a distributed filesystem as vertex and edge lists. - Each iteration (called a "superstep"): the framework calls a function for each vertex, passing it all messages sent to that vertex in the previous superstep. The vertex updates its local state and sends messages to neighboring vertices along graph edges. - Between supersteps, the framework delivers all messages: every message sent in superstep N is guaranteed to arrive at its destination vertex in superstep N+1. - A vertex that receives no messages and votes to halt is deactivated. The algorithm completes when all vertices are inactive. - Vertices communicate only by message passing, not by querying each other directly — this enables batching and provides a clean fault tolerance boundary. **Fault tolerance in Pregel:** At the end of each superstep, the framework checkpoints the full state of all vertices to durable storage. If a node fails, the computation rolls back to the last checkpoint and resumes. Because operators are deterministic and messages are logged, selective partition recovery is also possible. **When Pregel is appropriate:** - Graph is too large to fit in memory on a single machine - Algorithm requires many iterations over the graph structure - Examples: PageRank, single-source shortest path, strongly connected components, community detection **When Pregel is NOT appropriate:** - Graph fits on a single machine — a single-threaded algorithm will likely outperform a distributed one due to cross-machine communication overhead in distributed graph processing - Graph fits on a single machine's disks — frameworks like GraphChi enable single-machine processing of graphs larger than RAM - Algorithm is not naturally expressed as vertex-centric message passing **Implementations:** Apache Giraph, Spark GraphX, Flink Gelly API. **SKIP THIS STEP** if the pipeline does not operate on graph-structured data. Mark Step 5 complete in TodoWrite. --- ### Step 6: Design the Fault Tolerance Strategy **ACTION:** Specify how the pipeline recovers from task failures, job failures, and node failures. **WHY:** Batch jobs process large datasets over long runtimes, making failures statistically inevitable. A pipeline that cannot recover from partial failures forces full restarts — wasting hours of compute. The key insight is that fault tolerance in batch processing is much simpler than in online systems because inputs are immutable: a failed task can always be retried on the same input without risk of double-writes or corrupted state. The challenge is minimizing the work lost when a failure occurs. **Fault tolerance mechanisms:** 1. **Task-level retry (MapReduce and dataflow engines):** If an individual map or reduce task fails, the framework automatically retries it on another machine using the same input. This is safe because inputs are immutable and task outputs from failed attempts are discarded — only the successful attempt's output counts. This handles transient failures (network glitches, temporary disk errors) transparently. 2. **Materialization to durable storage (MapReduce):** Because every job's output is written to HDFS before downstream jobs start, a failed job can be restarted at that job's boundary without recomputing upstream jobs. For a 50-job pipeline, a failure in job 35 requires rerunning only jobs 35-50, not the entire pipeline. 3. **Lineage-based recomputation (Spark):** Spark tracks the lineage of each Resilient Distributed Dataset — which input partitions were used and which operators were applied. If a partition is lost due to node failure, it is recomputed from the lineage graph using the original input data. No intermediate state needs to be persisted to HDFS. Trade-off: recomputation is expensive if the lineage chain is long; checkpoint RDDs periodically for pipelines with many transformation stages. 4. **Operator checkpointing (Flink):** Flink periodically checkpoints operator state, allowing recovery to the last checkpoint without recomputing from original inputs. Particularly important for stateful operators (running aggregations, graph iteration state). 5. **Determinism requirement:** Recomputation is only correct if operators are deterministic — given the same input, they produce the same output. Non-deterministic behaviors to avoid: using system clocks in computations, iterating over hash tables (ordering is not guaranteed in most languages), using random numbers without a fixed seed, reading from external mutable sources during the job. **Selecting the fault tolerance strategy:** | Scenario | Recommendation | |----------|----------------| | Job runtime < 30 minutes | In-memory recomputation (Spark lineage); full materialization is overkill | | Job runtime > 1 hour with many stages | Checkpoint intermediate state periodically; full restart from scratch is too expensive | | Multi-team pipeline (other teams read intermediate outputs) | Full materialization to named HDFS paths; allows independent debugging and reuse | | High preemption rate (shared cluster, low-priority batch) | MapReduce-style full materialization; task-level retry handles frequent kills efficiently | | Iterative graph algorithm | Pregel checkpoint at each superstep boundary | Mark Step 6 complete in TodoWrite. --- ### Step 7: Specify Output Destination and Loading Strategy **ACTION:** Define where the pipeline writes its final output and how that output is safely loaded for serving. **WHY:** Writing pipeline output directly to a production database record-by-record is an anti-pattern: it is orders of magnitude slower than batch throughput, it overwhelms the database with parallel writes from hundreds of tasks, and it introduces partial-write visibility (consumers may read incomplete results mid-job). The correct pattern is to build the output as immutable files inside the batch job, then load them atomically. **Output patterns:** 1. **Distributed filesystem (HDFS, S3, GCS):** Write output to a new directory with an atomic rename at the end. Downstream jobs read the directory by name. This enables rollback (keep the previous directory; switch back if the new output is wrong) and supports multiple consumers. Preferred for intermediate pipeline outputs and for outputs consumed by other batch jobs. 2. **Search index build:** Mappers partition documents by shard, each reducer builds the index for its partition, and index files are written to the distributed filesystem. Index files are then atomically swapped into the search cluster (Lucene/Solr, Elasticsearch). The batch job produces a complete, correct index; the serving layer does a single atomic switch. 3. **Key-value store bulk load:** Build the key-value store files inside the batch job (the mapper extracts keys and sorts by key, which is already most of the work for building an index). Write the database files to the distributed filesystem. Load them into the serving system (Voldemort, HBase, Terrapin, ElephantDB) via bulk load, which copies files and atomically switches the server to query the new files. The old files remain available for rollback. 4. **Anti-pattern — writing to a live database per-record:** Do not use a database client library inside a mapper or reducer to write records one at a time to a production database. Reasons: (a) network round-trip per record kills throughput, (b) parallel tasks overwhelm the database's write capacity, (c) job failure leaves partial results visible to consumers, breaking the all-or-nothing guarantee. **The all-or-nothing guarantee:** A batch job's output is only visible when the job completes successfully. If the job fails, no output is produced. This property is what makes batch outputs safe to depend on. Preserving it for external outputs requires writing to a temporary location and atomically promoting the output on success. Mark Step 7 complete in TodoWrite. --- ### Step 8: Produce the Pipeline Architecture Document **ACTION:** Write a structured pipeline architecture document covering all decisions from Steps 1-7. **WHY:** The architecture document captures not just what the pipeline does but why each design decision was made. This is essential for future engineers who will modify the pipeline — knowing the reasoning prevents undoing sound decisions. It also serves as the basis for the implementation plan. **HANDOFF TO HUMAN:** The agent produces the architecture document; the human reviews trade-offs, confirms constraints, and proceeds to implementation. **Output format:** ```markdown # Batch Pipeline Architecture: [Pipeline Name] ## Workload Classification - Type: Batch (bounded input, periodic execution) - Input: [datasets, sizes, formats] - Output: [destination, format, consumers] - Latency tolerance: [acceptable job duration] ## Processing Engine - Selected: [MapReduce / Spark / Flink / Tez] - Rationale: [key factors from decision framework] ## Pipeline Stages (DAG) [Stage 1] → [Stage 2] → [Stage 3] → [Output] ### Stage N: [Name] - Input: [source dataset or upstream stage output] - Transformation: [what this stage does] - Output: [format, partitioning key] - WHY this is a separate stage: [reason] ## Join Strategy - Join type: [sort-merge / broadcast hash / partitioned hash] - Datasets joined: [A (size) × B (size)] - Selection rationale: [why this strategy fits the data characteristics] - Skew handling: [if applicable] ## Intermediate State Management - Strategy: [full materialization / pipelining / hybrid] - Checkpoint points: [where state is durably saved] ## Graph Processing (if applicable) - Model: [Pregel/BSP, implementation] - Algorithm: [what graph computation is being performed] - Iteration convergence: [how the algorithm knows when to stop] ## Fault Tolerance - Task failure: [retry policy] - Job failure: [restart boundary — which stages must rerun] - Determinism: [operator determinism guarantees] ## Output Loading - Pattern: [filesystem / search index build / key-value bulk load] - Atomicity mechanism: [how partial output is prevented from being visible] - Rollback strategy: [how to revert to previous output if new output is wrong] ## Trade-offs and Risks - [Key trade-off 1]: [why this decision was made, what is given up] - [Risk 1]: [failure mode and mitigation] ``` Mark Step 8 complete in TodoWrite. ## Inputs - **Data source descriptions:** Dataset names, sizes, formats, update frequency - **Required transformations:** Join conditions, aggregations, filters, enrichment steps - **Output destination:** Where results go, how they will be consumed - **Latency tolerance:** Acceptable job completion time - **Infrastructure constraints:** Existing cluster type, available tools, team expertise ## Outputs - **Pipeline architecture document** — engine choice, DAG of stages, join strategies, fault tolerance, output loading pattern - **Join strategy selection** — which of the three join algorithms applies and why - **Engine recommendation** — MapReduce vs Spark/Flink/Tez with decision rationale - **Fault tolerance design** — materialization vs recomputation checkpointing strategy ## Key Principles - **Bring data to the computation, not computation to the data** — the reason for co-locating datasets in a distributed filesystem before joining is that random remote lookups (querying a database per record) cost orders of magnitude more than local reads. MapReduce handles all network communication; the application code only sees local data. This separation also shields application code from partial failures. - **Treat inputs as immutable and outputs as complete-or-nothing** — immutable inputs make retries safe (a failed task reruns on the same input, cannot corrupt it). Complete-or-nothing outputs make pipelines composable and debuggable (a job either produced a correct output or produced no output; there is no third state). These two properties together enable fault tolerance without distributed transactions. - **Materialization is the trade-off between durability and performance** — fully materializing intermediate state to HDFS maximizes durability and restartability at the cost of extra I/O. Keeping state in memory maximizes performance at the cost of recomputation on failure. The right balance depends on job duration, cluster reliability, and whether intermediate outputs have other consumers. - **Join strategy depends on data layout, not just data size** — the fastest join is not always the one that handles the largest data; it is the one that matches the data's existing partitioning. A broadcast hash join is fastest when one table is small. A partitioned hash join is fastest when both tables are co-partitioned. A sort-merge join is always correct but always pays the shuffle cost. Knowing what assumptions you can make about data layout unlocks faster strategies. - **Skew breaks the "work is evenly distributed" assumption** — most batch processing performance analysis assumes uniform key distribution. Hot keys (celebrities, viral content, common stop words) concentrate work on a single reducer, making that reducer the bottleneck for the entire job. Identify potential hot keys during design and plan skew handling explicitly. - **Graph algorithms on distributed systems pay a high communication cost** — distributed graph processing sends messages across the network for every edge crossing a machine boundary. If a graph fits on one machine, a single-machine algorithm almost always outperforms a distributed one. Use distributed graph processing (Pregel) only when the graph genuinely cannot fit on a single machine or disk. - **Write output as files, bulk-load atomically into serving systems** — building the serving database inside the batch job (as sorted, indexed files on the distributed filesystem) and loading it atomically produces correct, rollback-able results. Writing per-record to a live database during the job trades away the all-or-nothing guarantee for implementation convenience, which is a bad trade. ## Examples **Scenario: Web server log analysis — top URLs by traffic** Trigger: "We have 500 GB of daily nginx access logs in S3. We need a daily report of the top 1000 most-requested URLs." Process: 1. Workload: batch — daily job, logs are a bounded daily dump, 24-hour latency acceptable. 2. Engine: Spark — single-stage job, but team already has Spark infrastructure; no reason to use raw MapReduce. 3. Stage 1 (map): Parse log lines, extract URL field. Emit (url, 1) pairs. 4. Stage 2 (reduce): Group by URL, sum counts. Emit (url, count). 5. Stage 3 (sort): Sort by count descending, take top 1000. 6. No joins required — single dataset. 7. Output: Write top-1000 list to S3 as a JSON file. Atomic rename prevents partial reads. 8. Fault tolerance: Spark lineage — job is short enough (< 30 min) that full recomputation from S3 is acceptable if a task fails. Output: Architecture doc specifying a 3-operator Spark pipeline with no joins, output to S3 JSON, daily schedule via Airflow. --- **Scenario: User activity enrichment — joining events with profile database** Trigger: "We have 2 TB of daily clickstream events (user_id, timestamp, url) and a 50 GB user profile database snapshot (user_id, age, country). We need to produce a dataset of (url, viewer_age, viewer_country) for analytics." Process: 1. Workload: batch — daily job, both datasets are fixed snapshots, latency up to 6 hours acceptable. 2. Engine: Spark — multi-stage workflow benefits from dataflow execution. 3. Join strategy: broadcast hash join — user profile database (50 GB) is small enough to broadcast to each executor; 2 TB clickstream is the large dataset. Each executor loads profile hash table at startup, scans clickstream records, looks up profile by user_id. No shuffle required. 4. Skew check: if any user_id has pathologically many events (bot traffic), filter or handle separately — but for typical users, assume uniform distribution. 5. Stage 1: Load both datasets. Broadcast profile table. 6. Stage 2 (map-side join): For each clickstream record, look up user profile by user_id. Emit (url, age, country). 7. Stage 3: Write output partitioned by date to HDFS. 8. Output: HDFS directory, Parquet format, read by Hive/Spark SQL for analytics queries. 9. Fault tolerance: Spark lineage; profile table is fully in memory per executor so recomputation is fast. Output: Architecture doc specifying broadcast hash join, Spark execution, Parquet output to HDFS, with note that if profile table grows beyond 200 GB, re-evaluate to sort-merge join. --- **Scenario: PageRank computation on a web link graph** Trigger: "We have a 10 TB graph of web pages and hyperlinks (300 billion edges). We need to compute PageRank scores for all pages to power search result ranking." Process: 1. Workload: batch — graph is a periodic snapshot, computation may take hours to days, latency not critical. 2. Engine: Spark GraphX (implements Pregel/BSP model) — algorithm is iterative and graph-structured; plain MapReduce would re-read 10 TB per iteration. 3. Graph processing: Pregel model. Each vertex holds its current PageRank score. Each superstep: vertex sends its score divided by out-degree to all neighbors. Vertex receives scores from all incoming neighbors and updates its own score as weighted sum. Repeat until convergence (change in scores below threshold, typically 0.001). 4. Fault tolerance: Checkpoint vertex state to HDFS every 5 supersteps (configurable). On node failure, roll back to last checkpoint and resume. Algorithm is deterministic, so checkpointing + rollback is safe. 5. Skew: PageRank is a "pull" algorithm — vertices pull scores from neighbors. High-degree "celebrity" pages receive many messages; this is handled by the framework's partitioning, not by skew join handling. 6. Single-machine feasibility check: 10 TB graph — does not fit on a single machine. Distributed processing is required. 7. Output: Write (page_id, pagerank_score) to HDFS. Bulk-load into serving key-value store for search ranking lookups. Output: Architecture doc specifying GraphX Pregel implementation, 5-superstep checkpoint interval, bulk load to serving store, with note to evaluate GraphChi if graph shrinks to single-machine scale. ## References - For detailed join algorithm decision criteria and size thresholds, see [join-strategy-reference.md](references/join-strategy-reference.md) - For MapReduce vs dataflow engine comparison with benchmark data, see [engine-comparison.md](references/engine-comparison.md) - For skew handling patterns (skewed join, sharded join, two-stage aggregation), see [join-strategy-reference.md](references/join-strategy-reference.md) - Cross-reference: for stream processing (unbounded input), see `stream-processing-designer` - Cross-reference: for workload classification (batch vs OLTP vs OLAP), see `oltp-olap-workload-classifier` - Source: *Designing Data-Intensive Applications*, Martin Kleppmann, Chapter 10 (pages 389-430) ## License This skill is licensed under [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/). Source: [BookForge](https://github.com/bookforge-ai/bookforge-skills) — Designing Data Intensive Applications by Unknown. ## Related BookForge Skills Install related skills from ClawhHub: - `clawhub install bookforge-oltp-olap-workload-classifier` Or install the full book set from GitHub: [bookforge-skills](https://github.com/bookforge-ai/bookforge-skills)

path: references/engine-comparison.md
content: # Batch Processing Engine Comparison

Detailed comparison of MapReduce, Spark, Flink, and Tez for batch pipeline selection.

## The Core Distinction: Materialization vs Pipelining

The most important difference between MapReduce and dataflow engines is how they handle data flow between stages.

**MapReduce — full materialization:**
- Every job writes all output to HDFS before the next job starts.
- The next job reads from HDFS.
- Intermediate data is replicated across machines (durability overhead).
- Each stage waits for 100% of the previous stage to complete before starting.
- Straggler tasks (slow tasks in a stage) delay the entire pipeline.

**Dataflow engines (Spark, Flink, Tez) — pipelining:**
- Intermediate state between operators is kept in memory or written to local disk.
- Operators start processing as soon as upstream operators produce output.
- No HDFS replication of intermediate data.
- No waiting for full stage completion before downstream starts.
- Fault recovery requires recomputation (Spark lineage) or checkpoint rollback (Flink).

---

## MapReduce

**Strengths:**
- Extremely robust — designed to run on preemptible, low-priority compute alongside production workloads.
- Task-level fault tolerance: failed tasks are retried on the same input; output from failed attempts is discarded.
- Simple programming model — mapper and reducer callbacks are the entire API.
- Mature tooling, extensive operational experience.
- Correct for any batch workload; no assumptions about data layout or memory.

**Weaknesses:**
- Multi-stage pipelines require full materialization of each stage to HDFS — expensive for complex workflows.
- Sorting is always performed between map and reduce, even when not needed.
- A new JVM process is launched per task — startup overhead adds up for short tasks.
- Cannot express arbitrary operator graphs; only map-then-reduce topology (mitigated by chaining jobs).

**When MapReduce is the right choice:**
- Cluster runs mixed workloads with high task preemption rates (Google-style environment).
- Job must tolerate very frequent task kills without any state loss.
- Simple 1-3 stage pipelines where materialization overhead is not significant.
- Existing Hadoop-only environment with no Spark/Flink infrastructure.
- Compliance/audit requirements demand fully durable intermediate state.

**Typical performance vs Spark:** 2-10x slower for multi-stage pipelines. Comparable for single-stage jobs.

---

## Apache Spark

**Strengths:**
- Significantly faster than MapReduce for multi-stage pipelines — intermediate data stays in memory or local disk.
- Resilient Distributed Dataset (RDD) abstraction tracks data lineage for fault recovery without HDFS materialization.
- Rich API: batch (RDD, DataFrame), SQL (Spark SQL), streaming (Structured Streaming), ML (MLlib), graph (GraphX).
- Spark SQL optimizer (Catalyst) automatically selects join algorithms (broadcast, sort-merge) based on table statistics.
- Strong ecosystem: PySpark, Scala, Java, R APIs; deep integration with Delta Lake, Iceberg, Hudi.

**Weaknesses:**
- Memory-intensive: keeping intermediate state in memory requires careful memory tuning.
- Long lineage chains accumulate recomputation cost on failure — checkpoint RDDs for very long pipelines.
- Non-deterministic operations (system time, Python hash randomization, external reads) make lineage-based recovery unsafe without explicit handling.
- Higher operational complexity than MapReduce.

**When Spark is the right choice:**
- Multi-stage pipelines (4+ stages) where MapReduce would materialize to HDFS between each.
- Machine learning pipelines (MLlib integration).
- Iterative algorithms (graph processing via GraphX, ML training).
- Mixed batch + near-real-time processing (Structured Streaming runs the same code on streams).
- Teams already familiar with Spark; rich Python (PySpark) ecosystem.

**Fault tolerance model:** Lineage tracking. Each RDD/DataFrame partition knows its lineage — which operators and input partitions produced it. On node failure, lost partitions are recomputed from lineage. For very long lineage chains (many transformation stages), periodically call `rdd.checkpoint()` to materialize to HDFS and truncate the lineage.

**Key configuration levers:**
- `spark.executor.memory`: memory per executor — must accommodate working set of intermediate data.
- `spark.sql.autoBroadcastJoinThreshold`: maximum table size for automatic broadcast hash join (default 10 MB; increase if small table fits in memory).
- `spark.default.parallelism`: number of partitions for RDD operations.

---

## Apache Flink

**Strengths:**
- Lowest latency among dataflow engines — pipelined execution (Flink does not wait for a stage to complete before starting the next).
- Strongest streaming integration — same code runs on bounded (batch) and unbounded (stream) input; designed for eventual stream processing migration.
- Stateful operators with exactly-once guarantees via periodic checkpointing.
- Gelly API for graph processing (Pregel/BSP model).
- Lower memory footprint than Spark for many workloads (Flink manages memory outside JVM heap for serialized data).

**Weaknesses:**
- Smaller ecosystem than Spark; fewer pre-built ML libraries.
- More complex operational tuning (checkpoint configuration, state backend selection).
- Less SQL ecosystem maturity than Spark SQL (catching up with Flink SQL).

**When Flink is the right choice:**
- Batch pipeline that will evolve into a stream pipeline — same job graph works for both.
- Lowest latency batch execution is required (minimize time from job submission to output availability).
- Stateful batch computations (running aggregations, sessionization, iterative graph processing).
- Already running Flink for stream processing; use same cluster for batch.

**Fault tolerance model:** Periodic operator checkpointing. Flink snapshots the state of all operators to durable storage at configurable intervals. On node failure, Flink rolls back the entire job to the last checkpoint and resumes execution. Unlike Spark's lineage recomputation, Flink's recovery time is proportional to checkpoint interval, not lineage length.

**Key configuration levers:**
- `execution.checkpointing.interval`: how often to checkpoint (trade-off between recovery time and checkpoint overhead).
- `state.backend`: where checkpoint state is stored (RocksDB for large state, memory for small state).
- `taskmanager.memory.managed.fraction`: fraction of memory managed by Flink outside JVM heap.

---

## Apache Tez

**Strengths:**
- Thin optimization layer over YARN — minimal overhead, leverages existing YARN infrastructure.
- Enables Pig and Hive jobs to execute faster without rewriting job code (swap MapReduce execution engine for Tez via configuration).
- Good for organizations with large existing Pig/Hive codebases that need faster execution without migration.

**Weaknesses:**
- Does not provide its own high-level API — you use Tez through Pig, Hive, or Cascading.
- Relies on YARN shuffle service for data copying between nodes — less control over data transfer than Spark or Flink.
- Smaller community than Spark or Flink.

**When Tez is the right choice:**
- Existing Pig or Hive workflows need faster execution without code changes.
- Team is not ready to migrate to Spark/Flink but needs immediate performance improvement.
- Hadoop cluster with YARN is the only available infrastructure.

---

## High-Level Languages and Query Optimizers

In practice, engineers rarely write raw MapReduce, Spark, or Flink operator code for most batch workloads. High-level languages compile to the underlying engine and include query optimizers that automatically select the best execution plan.

| Language / Tool | Underlying engine | Notes |
|----------------|------------------|-------|
| Hive | MapReduce, Tez, or Spark | SQL-like; Hive query optimizer selects joins automatically |
| Pig | MapReduce or Tez | Dataflow scripting language |
| Spark SQL / DataFrames | Spark | Catalyst optimizer; column pruning, join reordering, predicate pushdown |
| Flink SQL | Flink | Growing SQL support; same optimizer principles |
| Cascading | MapReduce or Tez | Java API |
| Crunch | MapReduce | Java/Scala API |

The advantage of high-level declarative interfaces: you declare WHAT joins you need, and the query optimizer decides HOW to execute them (which join algorithm, in what order). Hive, Spark, and Flink all have cost-based optimizers that can even reorder joins to minimize intermediate state.

**Recommendation:** Use Spark SQL or Hive for most batch workloads unless fine-grained control over join algorithms or operator behavior is required. Reserve raw RDD/operator API for cases the SQL optimizer cannot handle (custom aggregations, iterative algorithms, graph processing).

---

## Performance Comparison Summary

For a representative 10-stage MapReduce pipeline on the same hardware:

| Engine | Relative speed | Primary reason |
|--------|---------------|----------------|
| MapReduce | 1x (baseline) | Full HDFS materialization between all stages |
| Tez | 2-4x | Eliminates unnecessary map stages; reduces HDFS I/O |
| Spark | 5-20x | In-memory intermediate state; pipelined execution; no sort unless needed |
| Flink | 5-20x | Pipelined execution; lower memory overhead than Spark |

Note: actual speedup depends heavily on the pipeline's bottleneck. Disk-bound pipelines see larger Spark/Flink improvements; CPU-bound pipelines see smaller differences. For single-stage jobs with no intermediate state, the difference is minimal.

---

## Engine Selection Decision Tree

```
Is the cluster shared with production workloads and task preemption is frequent?
├── YES → MapReduce (designed for this; tolerates frequent task kills)
└── NO
    └── Does an existing Hive/Pig codebase need faster execution without rewriting?
        ├── YES → Tez (swap execution engine via config)
        └── NO
            └── Will this pipeline eventually process streaming (unbounded) input?
                ├── YES → Flink (same code for batch + stream)
                └── NO
                    └── Is Python ecosystem important? (data science, ML)
                        ├── YES → Spark (PySpark, MLlib, GraphX)
                        └── NO → Spark or Flink (comparable; choose based on team familiarity)
```


path: references/join-strategy-reference.md
content: # Join Strategy Reference

Detailed decision criteria, size thresholds, and skew handling patterns for batch pipeline joins.

## The Three Join Strategies at a Glance

| Strategy | Also called | Where join logic runs | Input requirements | Best for |
|----------|-------------|----------------------|-------------------|----------|
| Sort-merge join | Reduce-side join | Reducer | No assumptions | Default; both datasets large |
| Broadcast hash join | Map-side join (replicated) | Mapper | One dataset fits in memory | One small + one large dataset |
| Partitioned hash join | Bucketed map join (Hive) | Mapper | Both co-partitioned identically | Both datasets large, same partitioning |

---

## Strategy 1: Sort-Merge Join (Reduce-Side)

### How it works

1. **Map phase:** Both input datasets go through mappers. Each mapper extracts the join key and emits `(join_key, record)` pairs. Records from dataset A and dataset B both emit using the same join key.
2. **Shuffle/sort phase:** The framework partitions mapper output by join key (using hash-of-key to assign to a reducer partition) and sorts within each partition. All records with the same join key — from both datasets — arrive at the same reducer, adjacent in sort order.
3. **Reduce phase:** The reducer iterates over all records with a given key. Using a secondary sort (dataset A records before dataset B records, or vice versa), the reducer can hold one dataset's record in memory while streaming the other, rather than buffering everything. The reducer outputs joined records.

### Secondary sort technique

Secondary sort arranges records so that the "small side" of the join (e.g., one profile record per user) arrives first. The reducer stores this record in a local variable, then iterates through "large side" records (e.g., many activity events per user) emitting a joined output for each. This keeps memory usage constant — only one record from the small side is held at a time.

### Size thresholds

- Both datasets can be arbitrarily large.
- Network and disk I/O scale with total data volume. Budget 2-3x the input data size for shuffle traffic.
- Optimize by filtering and projecting early in the map phase to reduce shuffle volume.

### When to use

- Default choice when dataset sizes are unknown or both are large (> a few GB each).
- When no assumptions can be made about data partitioning.
- When the join key is not known in advance to be uniformly distributed (safe starting point; add skew handling if needed).

### Cost profile

- Expensive: full shuffle of both datasets over the network, full sort.
- Reducer start is delayed until all mappers for both datasets have finished.
- Intermediate data is written to local disk on each mapper, then transferred to reducers.

---

## Strategy 2: Broadcast Hash Join (Map-Side, Small Table)

### How it works

1. Before map tasks start, the small dataset is loaded from the distributed filesystem into each mapper's memory as a hash table (keyed by join key).
2. Each mapper reads one block of the large dataset and, for each record, looks up the join key in the in-memory hash table.
3. The mapper emits joined records directly. No reducer needed. No shuffle.

### Size thresholds

- Small table must fit in memory on each mapper (including other mapper state and JVM overhead).
- Practical limits: 1-4 GB per mapper is typical. Check executor memory configuration.
- If the small table is larger than memory but can fit on local disk, some implementations (Pig "replicated join", Hive "MapJoin") can use a local disk index with OS page cache to achieve near-memory performance.

### Multi-way broadcast joins

If multiple small tables need to be joined with one large table, all small tables can be broadcast simultaneously. Each mapper loads all small tables at startup. This is efficient as long as the total memory footprint of all small tables stays within the mapper's memory budget.

### Supported in

- Pig: "replicated join"
- Hive: "MapJoin" (also auto-detected by Hive optimizer when table is below `hive.mapjoin.smalltable.filesize`)
- Cascading, Crunch: broadcast join operators
- Spark: `broadcast()` hint on small DataFrame; optimizer uses `BroadcastHashJoin`
- Impala: used by default for small tables in analytic queries

### When to use

- One dataset is clearly small (user profile DB, product catalog, lookup table) and the other is large (event log, transaction history).
- When small table is frequently joined with many large datasets (load once, reuse across multiple map tasks).

### When NOT to use

- Small table grows beyond memory budget — switch to sort-merge join.
- Multiple large-large joins — broadcast only one side; the other must be shuffled.

---

## Strategy 3: Partitioned Hash Join (Map-Side, Co-Partitioned)

### How it works

Both datasets must be partitioned using the same key, the same hash function, and the same number of partitions. This is the "co-partitioned" or "bucketed" property.

1. Mapper N reads partition N of the large dataset.
2. Mapper N also loads partition N of the small dataset into memory (only partition N, not the full dataset).
3. Mapper N performs a hash join between the two partitions locally. No shuffle or reducer needed.

Because the co-partitioning guarantee means that any record in partition N of dataset A can only match records in partition N of dataset B, each mapper handles a fully independent subset of the join problem.

### Size thresholds

- Neither dataset needs to fit entirely in memory — only one partition at a time.
- Memory requirement: (total small dataset size) / (number of partitions).
- With 1000 partitions and a 100 GB small dataset: each mapper loads 100 MB. Very manageable.
- Large dataset can be arbitrarily large as long as one partition fits in local disk.

### Prerequisites

Co-partitioning is a strong prerequisite. The datasets must:
- Be partitioned by the same join key (e.g., both partitioned by `user_id`)
- Use the same hash function
- Have the same number of partitions

This is most naturally satisfied when both datasets were produced by prior MapReduce jobs with the same output partitioning scheme. The partitioning metadata (key, function, partition count) must be tracked — Hadoop ecosystem uses HCatalog / Hive metastore for this.

### Known as

- Hive: "bucketed map join"
- General: "co-partitioned hash join", "partition hash join"

### When to use

- Both datasets are large, but one is smaller than the other.
- Both datasets were produced by prior pipeline stages that happen to partition by the join key.
- The join key is the same as the partitioning key already in use.

### When NOT to use

- Datasets are partitioned by different keys, different hash functions, or different partition counts — co-partitioning guarantee does not hold.
- One dataset is small enough for broadcast hash join (strategy 2 is simpler and faster).

---

## Map-Side Merge Join (Bonus Variant)

If both datasets are not only co-partitioned but also **sorted by the join key within each partition**, a mapper can perform a merge join by reading both partition files incrementally in ascending key order and matching records with the same key — exactly like the merge step of a merge sort.

This requires no in-memory hash table, making it suitable for very large partitions. It works like the reduce phase of a sort-merge join but runs in a mapper (no shuffle required).

This situation often arises naturally if the datasets were produced by prior sort-merge join outputs (which are partitioned and sorted by definition).

---

## Skew Handling Patterns

### When to suspect skew

- A social network with celebrity accounts
- Content platforms with viral items (a single video_id with 10x the events of the average)
- Natural language data (stop words like "the", "a" as join keys)
- Time-partitioned data where some time windows have disproportionate traffic

### Detecting skew

Run a preliminary sampling job (or query the dataset statistics if available) to estimate the top-N key frequencies. If the top key has 10x the frequency of the median key, skew handling is warranted.

### Pattern 1: Skewed Join (Pig)

1. **Sampling job:** Scan a sample of the large dataset to identify hot keys.
2. **Hot key handling:** Records with hot keys are sent to a randomly chosen subset of reducers (spreading the load). Records from the other dataset for those hot keys must be replicated to all reducers handling that key.
3. **Cold key handling:** Standard sort-merge join.

Cost: data for hot-key matching side is replicated (increases network volume). Benefit: the hot-key reducer becomes a set of reducers, eliminating the straggler.

### Pattern 2: Sharded Join (Crunch)

Same approach as skewed join but hot keys are specified explicitly by the developer rather than discovered by sampling. Requires domain knowledge of which keys are hot.

### Pattern 3: Two-Stage Aggregation for GROUP BY Skew

When grouping by a hot key (not joining), a two-stage approach avoids the straggler reducer:

1. **Stage 1:** Each mapper sends records to a randomly chosen reducer (not the canonical hash-of-key reducer). Each reducer computes a partial aggregate for its subset of records.
2. **Stage 2:** The partial aggregates for each key (now much smaller in volume) are sent to the canonical reducer for final aggregation.

This is only correct for commutative, associative aggregations (sum, count, min, max). It is not correct for median or other order-dependent aggregations.

### Hive skewed join optimization

Hive's approach: hot keys are declared in the table metadata. Records for hot keys are stored in separate files. At join time, the hot-key records are processed using a broadcast hash join (map-side) while cold-key records use a standard sort-merge join. The join strategy is automatically selected per-key based on the metadata.

---

## Join Output and Downstream Partitioning

The join strategy affects the output's partitioning, which matters for downstream pipeline stages:

- **Sort-merge join output:** Partitioned and sorted by the join key (reducer output). Downstream stages that need the data sorted by join key can skip re-sorting.
- **Broadcast hash join output:** Partitioned the same way as the large input dataset (since one map task per block of the large input). Not sorted by join key.
- **Partitioned hash join output:** Partitioned the same way as both input datasets.

Knowing the output partitioning of each stage enables downstream stages to choose faster join strategies (e.g., if the sort-merge output is already sorted, the next join on the same key can use a map-side merge join).


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)