Repository: clflushopt/tpchgen-rs
Pull request #58: Parallel output in tpchgen-cli (Nx faster, where N is number of cores)
Author: login: alamb
Created: 2025-03-21T14:40:28Z | Merged: 2025-03-25T11:31:14Z

- closes https://github.com/clflushopt/tpchgen-rs/issues/10
- closes https://github.com/clflushopt/tpchgen-rs/pull/34
- builds on https://github.com/clflushopt/tpchgen-rs/pull/54

Features:
1. Logging support via `-v` and `RUST_LOG`
2. Parallel generation for all tables
3. No new dependencies on the tpch core crate and minor changes

# Performance 🌶️ 
In single threaded mode, the generator can create 250MB/sec with each core which is 4GB/s on my 16 core laptop

This scales linearly with 

Here are performance measurements for SF=100 on my Mac M3 (with 16 cores)

| branch | speed | speed (writing to /dev/null) |
|--------|--------|--------|
| main | 6m1.984s | N/A
| this branch | 1m0.099s | 0m28.732s | 




```shell
time target/release/tpchgen-cli -s 100 --output-dir=/tmp/tpchdbgen-rs
```

It actually turns out this can fully saturate the disk bandwidth on my Mac M3 laptop which caps out at 2GB/sec


If I hard code the generator to just throw the data away rather than writing, it keeps all the cores busy and writes at a blistering 4GB/sec



Diff:
diff --git a/tpchgen-cli/Cargo.toml b/tpchgen-cli/Cargo.toml
index f824809..8e42230 100644
--- a/tpchgen-cli/Cargo.toml
+++ b/tpchgen-cli/Cargo.toml
@@ -7,3 +7,8 @@ authors = ["clflushopt", "alamb"]
 [dependencies]
 clap = { version = "4.5.32", features = ["derive"] }
 tpchgen = { path = "../tpchgen" }
+tokio = { version = "1.44.1", features = ["full"]}
+futures = "0.3.31"
+num_cpus = "1.0"
+log = "0.4.26"
+env_logger = "0.11.7"
\ No newline at end of file
diff --git a/tpchgen-cli/src/csv.rs b/tpchgen-cli/src/csv.rs
new file mode 100644
index 0000000..d1ad566
--- /dev/null
+++ b/tpchgen-cli/src/csv.rs
@@ -0,0 +1,51 @@
+//! Implementations of [`Source`] for generating data in TBL format
+use super::generate::Source;
+use std::io::Write;
+use tpchgen::csv::{
+    CustomerCsv, LineItemCsv, NationCsv, OrderCsv, PartCsv, PartSuppCsv, RegionCsv, SupplierCsv,
+};
+use tpchgen::generators::{
+    CustomerGenerator, LineItemGenerator, NationGenerator, OrderGenerator, PartGenerator,
+    PartSupplierGenerator, RegionGenerator, SupplierGenerator,
+};
+
+/// Define a Source that writes the table in CSV format
+macro_rules! define_csv_source {
+    ($SOURCE_NAME:ident, $GENERATOR_TYPE:ty, $FORMATTER:ty) => {
+        pub struct $SOURCE_NAME {
+            inner: $GENERATOR_TYPE,
+        }
+
+        impl $SOURCE_NAME {
+            pub fn new(inner: $GENERATOR_TYPE) -> Self {
+                Self { inner }
+            }
+        }
+
+        impl Source for $SOURCE_NAME {
+            fn create(self, mut buffer: Vec<u8>) -> Vec<u8> {
+                writeln!(&mut buffer, "{}", <$FORMATTER>::header())
+                    .expect("writing to memory is infallible");
+                for item in self.inner.iter() {
+                    let formatter = <$FORMATTER>::new(item);
+                    writeln!(&mut buffer, "{formatter}").expect("writing to memory is infallible");
+                }
+                buffer
+            }
+        }
+    };
+}
+
+// Define .csv sources for all tables
+define_csv_source!(NationCsvSource, NationGenerator<'static>, NationCsv);
+define_csv_source!(RegionCsvSource, RegionGenerator<'static>, RegionCsv);
+define_csv_source!(PartCsvSource, PartGenerator<'static>, PartCsv);
+define_csv_source!(SupplierCsvSource, SupplierGenerator<'static>, SupplierCsv);
+define_csv_source!(
+    PartSuppCsvSource,
+    PartSupplierGenerator<'static>,
+    PartSuppCsv
+);
+define_csv_source!(CustomerCsvSource, CustomerGenerator<'static>, CustomerCsv);
+define_csv_source!(OrderCsvSource, OrderGenerator<'static>, OrderCsv);
+define_csv_source!(LineItemCsvSource, LineItemGenerator<'static>, LineItemCsv);
diff --git a/tpchgen-cli/src/generate.rs b/tpchgen-cli/src/generate.rs
new file mode 100644
index 0000000..41d6869
--- /dev/null
+++ b/tpchgen-cli/src/generate.rs
@@ -0,0 +1,146 @@
+//! Parallel data generation: [`Source`] and [`Sink`] and [`generate_in_chunks`]
+//!
+//! These traits and function are used to generate data in parallel and write it to a sink
+//! in streaming fashion (chunks). This is useful for generating large datasets that don't fit in memory.
+
+use futures::StreamExt;
+use log::debug;
+use std::collections::VecDeque;
+use std::io;
+use std::sync::{Arc, Mutex};
+use tokio::task::JoinSet;
+
+/// Something that knows how to generate data into a buffer
+///
+/// For example, this is implemented for the different generators in the tpchgen
+/// crate
+pub trait Source: Send {
+    /// generates the data for this generator into the buffer, returning the buffer.
+    fn create(self, buffer: Vec<u8>) -> Vec<u8>;
+}
+
+/// Something that can write the contents of a buffer somewhere
+///
+/// For example, this is implemented for a file writer
+pub trait Sink: Send {
+    /// Write all data from the buffer to the sink
+    fn sink(&mut self, buffer: &[u8]) -> Result<(), io::Error>;
+
+    /// Complete and flush any remaining data from the sink
+    fn flush(self) -> Result<(), io::Error>;
+}
+
+/// Generates data in parallel from a series of [`Source`] and writes to a [`Sink`]
+///
+/// Each [`Source`] is a data generator that generates data directly into an in
+/// memory buffer.
+///
+/// This function will run the [`Source`]es in parallel using all available
+/// cores. Data is written to the [`Sink`] in the order of the [`Source`]es in
+/// the input iterator.
+///
+/// G: Generator
+/// I: Iterator<Item = G>
+/// S: Sink that writes buffers somewhere
+pub async fn generate_in_chunks<G, I, S>(mut sink: S, sources: I) -> Result<(), io::Error>
+where
+    G: Source + 'static,
+    I: Iterator<Item = G>,
+    S: Sink + 'static,
+{
+    let recycler = BufferRecycler::new();
+
+    // use all cores to make data
+    let num_tasks = num_cpus::get();
+    debug!("Generating with {num_tasks} parallel tasks");
+
+    // create a channel to communicate between the generator tasks and the writer task
+    let (tx, mut rx) = tokio::sync::mpsc::channel(num_tasks);
+
+    let sources_and_recyclers = sources.map(|generator| (generator, recycler.clone()));
+
+    // convert to an async stream to run on tokio
+    let mut stream = futures::stream::iter(sources_and_recyclers)
+        // each generator writes to a buffer
+        .map(async |(source, recycler)| {
+            let buffer = recycler.new_buffer(1024 * 1024 * 8);
+            // do the work in a task (on a different thread)
+            let mut join_set = JoinSet::new();
+            join_set.spawn(async move { source.create(buffer) });
+            // wait for the task to be done and return the result
+            join_set
+                .join_next()
+                .await
+                .expect("had one item")
+                .expect("join_next join is infallible unless task panics")
+        })
+        // run in parallel
+        .buffered(num_tasks)
+        .map(async |buffer| {
+            // send the buffer to the writer task, in order.
+
+            // Note we ignore errors writing because if the write errors it
+            // means the channel is closed / the program is exiting so there
+            // is nothing listening to send errors
+            if let Err(e) = tx.send(buffer).await {
+                debug!("Error sending buffer to writer: {e}");
+            }
+        });
+
+    // The writer task runs in a blocking thread to avoid blocking the async
+    // runtime. It reads from the channel and writes to the sink (doing File IO)
+    let captured_recycler = recycler.clone();
+    let writer_task = tokio::task::spawn_blocking(move || {
+        while let Some(buffer) = rx.blocking_recv() {
+            sink.sink(&buffer)?;
+            captured_recycler.return_buffer(buffer);
+        }
+        // No more input, flush the sink and return
+        sink.flush()
+    });
+
+    // drive the stream to completion
+    while let Some(write_task) = stream.next().await {
+        write_task.await; // sends the buffer to the writer task
+    }
+    drop(stream); // drop any stream references
+    drop(tx); // drop last tx reference to tell the writer it is done.
+
+    // wait for writer to finish
+    debug!("waiting for writer task to complete");
+    writer_task.await.expect("writer task panicked")
+}
+
+/// A simple buffer recycler to avoid allocating new buffers for each part
+///
+/// Clones share the same underlying recycler, so it is not thread safe
+#[derive(Debug, Clone)]
+struct BufferRecycler {
+    buffers: Arc<Mutex<VecDeque<Vec<u8>>>>,
+}
+
+impl BufferRecycler {
+    fn new() -> Self {
+        Self {
+            buffers: Arc::new(Mutex::new(VecDeque::new())),
+        }
+    }
+    /// return a new empty buffer, with size bytes capacity
+    fn new_buffer(&self, size: usize) -> Vec<u8> {
+        let mut buffers = self.buffers.lock().unwrap();
+        if let Some(mut buffer) = buffers.pop_front() {
+            buffer.clear();
+            if size > buffer.capacity() {
+                buffer.reserve(size - buffer.capacity());
+            }
+            buffer
+        } else {
+            Vec::with_capacity(size)
+        }
+    }
+
+    fn return_buffer(&self, buffer: Vec<u8>) {
+        let mut buffers = self.buffers.lock().unwrap();
+        buffers.push_back(buffer);
+    }
+}
diff --git a/tpchgen-cli/src/main.rs b/tpchgen-cli/src/main.rs
index 70ad5f3..d116845 100644
--- a/tpchgen-cli/src/main.rs
+++ b/tpchgen-cli/src/main.rs
@@ -10,22 +10,44 @@
 //! -T, --tables     Tables to generate data for
 //! -F, --format     Output format for the data (CSV or Parquet)
 //! -O, --output     Output directory for the generated data
-//
-// The main function is the entry point for the CLI and it uses the `clap` crate
-// to parse the command line arguments and then generate the data.
-
-// tpchgen-cli/src/main.rs
+//! -v, --verbose    Verbose output
+//!
+//! # Logging:
+//! Use the `-v` flag or `RUST_LOG` environment variable to control logging output.
+//!
+//! `-v` sets the log level to `info` and ignores the `RUST_LOG` environment variable.
+//!
+//! # Examples
+//! ```
+//! # see all info output
+//! tpchgen-cli -s 1 -v
+//!
+//! # same thing using RUST_LOG
+//! RUST_LOG=info tpchgen-cli -s 1
+//!
+//! # see all debug output
+//! RUST_LOG=debug tpchgen -s 1
+//! ```
+mod csv;
+mod generate;
+mod tbl;
+
+use crate::csv::*;
+use crate::generate::{generate_in_chunks, Sink, Source};
+use crate::tbl::*;
 use clap::{Parser, ValueEnum};
+use log::{debug, info, LevelFilter};
+use std::fmt::Display;
 use std::fs::{self, File};
 use std::io::{self, BufWriter, Write};
 use std::path::PathBuf;
-use tpchgen::csv::{
-    CustomerCsv, LineItemCsv, NationCsv, OrderCsv, PartCsv, PartSuppCsv, RegionCsv, SupplierCsv,
-};
+use std::time::Instant;
+use tpchgen::distribution::Distributions;
 use tpchgen::generators::{
     CustomerGenerator, LineItemGenerator, NationGenerator, OrderGenerator, PartGenerator,
     PartSupplierGenerator, RegionGenerator, SupplierGenerator,
 };
+use tpchgen::text::TextPool;
 
 #[derive(Parser)]
 #[command(name = "tpchgen")]
@@ -54,6 +76,10 @@ struct Cli {
     /// Output format: tbl, csv, parquet (default: tbl)
     #[arg(short, long, default_value = "tbl")]
     format: OutputFormat,
+
+    /// Verbose output (default: false)
+    #[arg(short, long, default_value_t = false)]
+    verbose: bool,
 }
 
 #[derive(Debug, Copy, Clone, PartialEq, Eq, PartialOrd, Ord, ValueEnum)]
@@ -68,6 +94,12 @@ enum Table {
     LineItem,
 }
 
+impl Display for Table {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        write!(f, "{}", self.name())
+    }
+}
+
 impl Table {
     fn name(&self) -> &'static str {
         match self {
@@ -90,14 +122,53 @@ enum OutputFormat {
     Parquet,
 }
 
-fn main() -> io::Result<()> {
+#[tokio::main]
+async fn main() -> io::Result<()> {
     // Parse command line arguments
     let cli = Cli::parse();
-    cli.main()
+    cli.main().await
+}
+
+/// macro to create a Cli function for generating a table
+///
+/// Arguments:
+/// $FUN_NAME: name of the function to create
+/// $TABLE: The [`Table`] to generate
+/// $GENERATOR: The generator type to use
+/// $TBL_SOURCE: The [`Source`] type to use for TBL format
+/// $CSV_SOURCE: The [`Source`] type to use for CSV format
+macro_rules! define_generate {
+    ($FUN_NAME:ident,  $TABLE:expr, $GENERATOR:ident, $TBL_SOURCE:ty, $CSV_SOURCE:ty) => {
+        async fn $FUN_NAME(&self) -> io::Result<()> {
+            let filename = self.output_filename($TABLE);
+            let (num_parts, parts) = self.parallel_target_part_count(&$TABLE);
+            let scale_factor = self.scale_factor;
+            info!("Writing table {} (SF={scale_factor}) to {filename}", $TABLE);
+            debug!("Generating {num_parts} parts in total");
+            let gens = parts
+                .into_iter()
+                .map(move |part| $GENERATOR::new(scale_factor, part, num_parts));
+            match self.format {
+                OutputFormat::Tbl => self.go(&filename, gens.map(<$TBL_SOURCE>::new)).await,
+                OutputFormat::Csv => self.go(&filename, gens.map(<$CSV_SOURCE>::new)).await,
+                // https://github.com/clflushopt/tpchgen-rs/issues/46
+                OutputFormat::Parquet => unimplemented!("Parquet support not yet implemented"),
+            }
+        }
+    };
 }
 
 impl Cli {
-    fn main(self) -> io::Result<()> {
+    async fn main(self) -> io::Result<()> {
+        if self.verbose {
+            // explicitly set logging to info / stdout
+            env_logger::builder().filter_level(LevelFilter::Info).init();
+            info!("Verbose output enabled (ignoring RUST_LOG environment variable)");
+        } else {
+            env_logger::init();
+            debug!("Logging configured from environment variables");
+        }
+
         // Create output directory if it doesn't exist
         fs::create_dir_all(&self.output_dir)?;
 
@@ -117,24 +188,90 @@ impl Cli {
             ]
         };
 
+        // force the creation of the distributions and text pool to so it doesn't
+        // get charged to the first table
+        let start = Instant::now();
+        debug!("Creating distributions and text pool");
+        Distributions::static_default();
+        TextPool::get_or_init_default();
+        let elapsed = start.elapsed();
+        info!("Created static distributions and text pools in {elapsed:?}");
+
         // Generate each table
         for table in tables {
             match table {
-                Table::Nation => self.generate_nation()?,
-                Table::Region => self.generate_region()?,
-                Table::Part => self.generate_part()?,
-                Table::Supplier => self.generate_supplier()?,
-                Table::PartSupp => self.generate_partsupp()?,
-                Table::Customer => self.generate_customer()?,
-                Table::Orders => self.generate_orders()?,
-                Table::LineItem => self.generate_lineitem()?,
+                Table::Nation => self.generate_nation().await?,
+                Table::Region => self.generate_region().await?,
+                Table::Part => self.generate_part().await?,
+                Table::Supplier => self.generate_supplier().await?,
+                Table::PartSupp => self.generate_partsupp().await?,
+                Table::Customer => self.generate_customer().await?,
+                Table::Orders => self.generate_orders().await?,
+                Table::LineItem => self.generate_lineitem().await?,
             }
         }
 
-        println!("Generation complete!");
+        info!("Generation complete!");
         Ok(())
     }
 
+    define_generate!(
+        generate_nation,
+        Table::Nation,
+        NationGenerator,
+        NationTblSource,
+        NationCsvSource
+    );
+    define_generate!(
+        generate_region,
+        Table::Region,
+        RegionGenerator,
+        RegionTblSource,
+        RegionCsvSource
+    );
+    define_generate!(
+        generate_part,
+        Table::Part,
+        PartGenerator,
+        PartTblSource,
+        PartCsvSource
+    );
+    define_generate!(
+        generate_supplier,
+        Table::Supplier,
+        SupplierGenerator,
+        SupplierTblSource,
+        SupplierCsvSource
+    );
+    define_generate!(
+        generate_partsupp,
+        Table::PartSupp,
+        PartSupplierGenerator,
+        PartSuppTblSource,
+        PartSuppCsvSource
+    );
+    define_generate!(
+        generate_customer,
+        Table::Customer,
+        CustomerGenerator,
+        CustomerTblSource,
+        CustomerCsvSource
+    );
+    define_generate!(
+        generate_orders,
+        Table::Orders,
+        OrderGenerator,
+        OrderTblSource,
+        OrderCsvSource
+    );
+    define_generate!(
+        generate_lineitem,
+        Table::LineItem,
+        LineItemGenerator,
+        LineItemTblSource,
+        LineItemCsvSource
+    );
+
     /// return the output filename for the given table
     fn output_filename(&self, table: Table) -> String {
         let extension = match self.format {
@@ -152,239 +289,112 @@ impl Cli {
         Ok(BufWriter::with_capacity(32 * 1024, file))
     }
 
-    fn generate_nation(&self) -> io::Result<()> {
-        let filename = self.output_filename(Table::Nation);
-        let writer = self.new_output_writer(&filename)?;
-
-        let generator = NationGenerator::new();
-        match self.format {
-            OutputFormat::Tbl => self.nation_tbl(writer, generator),
-            OutputFormat::Csv => self.nation_csv(writer, generator),
-            OutputFormat::Parquet => {
-                unimplemented!("Parquet output not yet implemented");
-            }
-        }
-    }
-
-    fn generate_region(&self) -> io::Result<()> {
-        let filename = self.output_filename(Table::Region);
-        let writer = self.new_output_writer(&filename)?;
-
-        let generator = RegionGenerator::new();
-        match self.format {
-            OutputFormat::Tbl => self.region_tbl(writer, generator),
-            OutputFormat::Csv => self.region_csv(writer, generator),
-            OutputFormat::Parquet => {
-                unimplemented!("Parquet output not yet implemented");
-            }
-        }
-    }
-
-    fn generate_part(&self) -> io::Result<()> {
-        let filename = self.output_filename(Table::Part);
-        let writer = self.new_output_writer(&filename)?;
-
-        let generator = PartGenerator::new(self.scale_factor, self.part, self.parts);
-        match self.format {
-            OutputFormat::Tbl => self.part_tbl(writer, generator),
-            OutputFormat::Csv => self.part_csv(writer, generator),
-            OutputFormat::Parquet => {
-                unimplemented!("Parquet output not yet implemented");
-            }
-        }
-    }
-
-    fn generate_supplier(&self) -> io::Result<()> {
-        let filename = self.output_filename(Table::Supplier);
-        let writer = self.new_output_writer(&filename)?;
-
-        let generator = SupplierGenerator::new(self.scale_factor, self.part, self.parts);
-        match self.format {
-            OutputFormat::Tbl => self.supplier_tbl(writer, generator),
-            OutputFormat::Csv => self.supplier_csv(writer, generator),
-            OutputFormat::Parquet => {
-                unimplemented!("Parquet output not yet implemented");
-            }
-        }
-    }
-
-    fn generate_partsupp(&self) -> io::Result<()> {
-        let filename = self.output_filename(Table::PartSupp);
-        let writer = self.new_output_writer(&filename)?;
-
-        let generator = PartSupplierGenerator::new(self.scale_factor, self.part, self.parts);
-        match self.format {
-            OutputFormat::Tbl => self.partsupp_tbl(writer, generator),
-            OutputFormat::Csv => self.partsupp_csv(writer, generator),
-            OutputFormat::Parquet => {
-                unimplemented!("Parquet output not yet implemented");
-            }
-        }
-    }
-
-    fn generate_customer(&self) -> io::Result<()> {
-        let filename = self.output_filename(Table::Customer);
-        let writer = self.new_output_writer(&filename)?;
-
-        let generator = CustomerGenerator::new(self.scale_factor, self.part, self.parts);
-        match self.format {
-            OutputFormat::Tbl => self.customer_tbl(writer, generator),
-            OutputFormat::Csv => self.customer_csv(writer, generator),
-            OutputFormat::Parquet => {
-                unimplemented!("Parquet output not yet implemented");
-            }
-        }
-    }
-
-    fn generate_orders(&self) -> io::Result<()> {
-        let filename = self.output_filename(Table::Orders);
-        let writer = self.new_output_writer(&filename)?;
-
-        let generator = OrderGenerator::new(self.scale_factor, self.part, self.parts);
-        match self.format {
-            OutputFormat::Tbl => self.orders_tbl(writer, generator),
-            OutputFormat::Csv => self.orders_csv(writer, generator),
-            OutputFormat::Parquet => {
-                unimplemented!("Parquet output not yet implemented");
-            }
+    /// Returns a list of "parts" (data generator chunks, not TPCH parts) to create
+    ///
+    /// Tuple returned is `(num_parts, part_list)`:
+    /// - num_parts is the total number of parts to generate
+    /// - part_list is the list of parts to generate (1 based)
+    fn parallel_target_part_count(&self, table: &Table) -> (i32, Vec<i32>) {
+        // parallel generation disabled if user specifies a part explicitly
+        if self.part != 1 || self.parts != 1 {
+            return (self.parts, vec![self.part]);
         }
-    }
-
-    fn generate_lineitem(&self) -> io::Result<()> {
-        let filename = self.output_filename(Table::LineItem);
-        let writer = self.new_output_writer(&filename)?;
 
-        let generator = LineItemGenerator::new(self.scale_factor, self.part, self.parts);
-        match self.format {
-            OutputFormat::Tbl => self.lineitem_tbl(writer, generator),
-            OutputFormat::Csv => self.lineitem_csv(writer, generator),
-            OutputFormat::Parquet => {
-                unimplemented!("Parquet output not yet implemented");
+        // Note use part=1, part_count=1 to calculate the total row count
+        // for the table
+        //
+        // Avg row size is an estimate of the average row size in bytes from the first 100 rows
+        // of the table in tbl format
+        let (avg_row_size_bytes, row_count) = match table {
+            Table::Nation => (88, 1),
+            Table::Region => (77, 1),
+            Table::Part => (
+                115,
+                PartGenerator::calculate_row_count(self.scale_factor, 1, 1),
+            ),
+            Table::Supplier => (
+                140,
+                SupplierGenerator::calculate_row_count(self.scale_factor, 1, 1),
+            ),
+            Table::PartSupp => (
+                148,
+                PartSupplierGenerator::calculate_row_count(self.scale_factor, 1, 1),
+            ),
+            Table::Customer => (
+                160,
+                CustomerGenerator::calculate_row_count(self.scale_factor, 1, 1),
+            ),
+            Table::Orders => (
+                114,
+                OrderGenerator::calculate_row_count(self.scale_factor, 1, 1),
+            ),
+            Table::LineItem => {
+                // there are on average 4 line items per order.
+                // For example, in SF=10,
+                // * orders has 15,000,000 rows
+                // * lineitem has around 60,000,000 rows
+                let row_count = 4 * OrderGenerator::calculate_row_count(self.scale_factor, 1, 1);
+                (128, row_count)
             }
-        }
-    }
-
-    // Separate functions for each table/output format combination
-    // to ensure they are inlined / a single function doesn't get out of hand
-    // TODO: make these via macros
-
-    fn nation_tbl<W: Write>(&self, mut w: W, gen: NationGenerator) -> io::Result<()> {
-        for item in gen.iter() {
-            writeln!(&mut w, "{item}")?;
-        }
-        w.flush()
-    }
-
-    fn nation_csv<W: Write>(&self, mut w: W, gen: NationGenerator) -> io::Result<()> {
-        writeln!(w, "{}", NationCsv::header())?;
-        for item in gen.iter() {
-            writeln!(&mut w, "{}", NationCsv::new(item))?;
-        }
-        w.flush()
-    }
-
-    fn region_tbl<W: Write>(&self, mut w: W, gen: RegionGenerator) -> io::Result<()> {
-        for item in gen.iter() {
-            writeln!(&mut w, "{item}")?;
-        }
-        w.flush()
-    }
-
-    fn region_csv<W: Write>(&self, mut w: W, gen: RegionGenerator) -> io::Result<()> {
-        writeln!(w, "{}", RegionCsv::header())?;
-        for item in gen.iter() {
-            writeln!(&mut w, "{}", RegionCsv::new(item))?;
-        }
-        w.flush()
-    }
-
-    fn part_tbl<W: Write>(&self, mut w: W, gen: PartGenerator) -> io::Result<()> {
-        for item in gen.iter() {
-            writeln!(&mut w, "{item}")?;
-        }
-        w.flush()
-    }
-
-    fn part_csv<W: Write>(&self, mut w: W, gen: PartGenerator) -> io::Result<()> {
-        writeln!(w, "{}", PartCsv::header())?;
-        for item in gen.iter() {
-            writeln!(&mut w, "{}", PartCsv::new(item))?;
-        }
-        w.flush()
-    }
-
-    fn supplier_tbl<W: Write>(&self, mut w: W, gen: SupplierGenerator) -> io::Result<()> {
-        for item in gen.iter() {
-            writeln!(&mut w, "{item}")?;
-        }
-        w.flush()
-    }
-
-    fn supplier_csv<W: Write>(&self, mut w: W, generator: SupplierGenerator) -> io::Result<()> {
-        writeln!(w, "{}", SupplierCsv::header())?;
-        for item in generator.iter() {
-            writeln!(&mut w, "{}", SupplierCsv::new(item))?;
-        }
-        w.flush()
-    }
-
-    fn partsupp_tbl<W: Write>(&self, mut w: W, gen: PartSupplierGenerator) -> io::Result<()> {
-        for item in gen.iter() {
-            writeln!(&mut w, "{item}")?;
-        }
-        w.flush()
+        };
+        // target chunks of about 16MB (use 15MB to ensure we don't exceed the target size)
+        let target_chunk_size_bytes = 15 * 1024 * 1024;
+        let num_parts = ((row_count * avg_row_size_bytes) / target_chunk_size_bytes) + 1;
+        // convert to i32
+        let num_parts = num_parts.try_into().unwrap();
+        // generating all the parts
+        (num_parts, (1..=num_parts).collect())
     }
 
-    fn partsupp_csv<W: Write>(&self, mut w: W, gen: PartSupplierGenerator) -> io::Result<()> {
-        writeln!(w, "{}", PartSuppCsv::header())?;
-        for item in gen.iter() {
-            writeln!(&mut w, "{}", PartSuppCsv::new(item))?;
-        }
-        w.flush()
+    /// Generates the output file from the sources
+    async fn go<I>(&self, filename: &str, sources: I) -> Result<(), io::Error>
+    where
+        I: Iterator<Item: Source> + 'static,
+    {
+        let sink = BufWriterSink::new(self.new_output_writer(filename)?);
+        generate_in_chunks(sink, sources).await
     }
+}
 
-    fn customer_tbl<W: Write>(&self, mut w: W, gen: CustomerGenerator) -> io::Result<()> {
-        for item in gen.iter() {
-            writeln!(&mut w, "{item}")?;
-        }
-        w.flush()
-    }
+/// Wrapper around a buffer writer that counts the number of buffers and bytes written
+struct BufWriterSink {
+    start: Instant,
+    inner: BufWriter<File>,
+    num_buffers: usize,
+    num_bytes: usize,
+}
 
-    fn customer_csv<W: Write>(&self, mut w: W, gen: CustomerGenerator) -> io::Result<()> {
-        writeln!(w, "{}", CustomerCsv::header())?;
-        for item in gen.iter() {
-            writeln!(&mut w, "{}", CustomerCsv::new(item))?;
+impl BufWriterSink {
+    fn new(inner: BufWriter<File>) -> Self {
+        Self {
+            start: Instant::now(),
+            inner,
+            num_buffers: 0,
+            num_bytes: 0,
         }
-        w.flush()
     }
+}
 
-    fn orders_tbl<W: Write>(&self, mut w: W, gen: OrderGenerator) -> io::Result<()> {
-        for item in gen.iter() {
-            writeln!(&mut w, "{item}")?;
-        }
-        w.flush()
+impl Sink for BufWriterSink {
+    fn sink(&mut self, buffer: &[u8]) -> Result<(), io::Error> {
+        self.num_buffers += 1;
+        self.num_bytes += buffer.len();
+        self.inner.write_all(buffer)
     }
 
-    fn orders_csv<W: Write>(&self, mut w: W, gen: OrderGenerator) -> io::Result<()> {
-        writeln!(w, "{}", OrderCsv::header())?;
-        for item in gen.iter() {
-            writeln!(&mut w, "{}", OrderCsv::new(item))?;
-        }
-        w.flush()
-    }
+    fn flush(mut self) -> Result<(), io::Error> {
+        let res = self.inner.flush();
 
-    fn lineitem_tbl<W: Write>(&self, mut w: W, gen: LineItemGenerator) -> io::Result<()> {
-        for item in gen.iter() {
-            writeln!(&mut w, "{item}")?;
-        }
-        w.flush()
-    }
+        let duration = self.start.elapsed();
+        let mb_per_buffer = self.num_bytes as f64 / (1024.0 * 1024.0) / self.num_buffers as f64;
+        let bytes_per_second = (self.num_bytes as f64 / duration.as_secs_f64()) as u64;
+        let gb_per_second = bytes_per_second as f64 / (1024.0 * 1024.0 * 1024.0);
 
-    fn lineitem_csv<W: Write>(&self, mut w: W, gen: LineItemGenerator) -> io::Result<()> {
-        writeln!(w, "{}", LineItemCsv::header())?;
-        for item in gen.iter() {
-            writeln!(&mut w, "{}", LineItemCsv::new(item))?;
-        }
-        w.flush()
+        info!("Completed in {duration:?} ({gb_per_second:.02} GB/sec)");
+        debug!(
+            "wrote {} bytes in {} buffers {mb_per_buffer:.02} MB/buffer",
+            self.num_bytes, self.num_buffers,
+        );
+        res
     }
 }
diff --git a/tpchgen-cli/src/tbl.rs b/tpchgen-cli/src/tbl.rs
new file mode 100644
index 0000000..701f8fd
--- /dev/null
+++ b/tpchgen-cli/src/tbl.rs
@@ -0,0 +1,43 @@
+//! Implementations of [`Source`] for generating data in TBL format
+
+use super::generate::Source;
+use std::io::Write;
+use tpchgen::generators::{
+    CustomerGenerator, LineItemGenerator, NationGenerator, OrderGenerator, PartGenerator,
+    PartSupplierGenerator, RegionGenerator, SupplierGenerator,
+};
+
+/// Define a Source that writes the table in TBL format
+macro_rules! define_tbl_source {
+    ($SOURCE_NAME:ident, $GENERATOR_TYPE:ty) => {
+        pub struct $SOURCE_NAME {
+            inner: $GENERATOR_TYPE,
+        }
+
+        impl $SOURCE_NAME {
+            pub fn new(inner: $GENERATOR_TYPE) -> Self {
+                Self { inner }
+            }
+        }
+
+        impl Source for $SOURCE_NAME {
+            fn create(self, mut buffer: Vec<u8>) -> Vec<u8> {
+                for item in self.inner.iter() {
+                    // The default Display impl writes TBL format
+                    writeln!(&mut buffer, "{item}").expect("writing to memory is infallible");
+                }
+                buffer
+            }
+        }
+    };
+}
+
+// Define .tbl sources for all tables
+define_tbl_source!(NationTblSource, NationGenerator<'static>);
+define_tbl_source!(RegionTblSource, RegionGenerator<'static>);
+define_tbl_source!(PartTblSource, PartGenerator<'static>);
+define_tbl_source!(SupplierTblSource, SupplierGenerator<'static>);
+define_tbl_source!(PartSuppTblSource, PartSupplierGenerator<'static>);
+define_tbl_source!(CustomerTblSource, CustomerGenerator<'static>);
+define_tbl_source!(OrderTblSource, OrderGenerator<'static>);
+define_tbl_source!(LineItemTblSource, LineItemGenerator<'static>);
diff --git a/tpchgen/src/csv.rs b/tpchgen/src/csv.rs
index c7d1f9c..43a0e0f 100644
--- a/tpchgen/src/csv.rs
+++ b/tpchgen/src/csv.rs
@@ -12,7 +12,7 @@ use std::fmt::Display;
 /// # use tpchgen::csv::NationCsv;
 /// # use std::fmt::Write;
 /// // Output the first 3 rows in CSV format
-/// let generator = NationGenerator::new();
+/// let generator = NationGenerator::default();
 /// let mut csv = String::new();
 /// writeln!(&mut csv, "{}", NationCsv::header()).unwrap(); // write header
 /// for line in generator.iter().take(3) {
@@ -60,7 +60,7 @@ impl Display for NationCsv<'_> {
 /// # use tpchgen::generators::{RegionGenerator};
 /// # use tpchgen::csv::{NationCsv, RegionCsv};
 /// # use std::fmt::Write;
-/// let generator = RegionGenerator::new();
+/// let generator = RegionGenerator::default();
 /// // Output the first 3 rows in CSV format
 /// let mut csv = String::new();
 /// writeln!(&mut csv, "{}", RegionCsv::header()).unwrap(); // write header
diff --git a/tpchgen/src/generators.rs b/tpchgen/src/generators.rs
index 98e5588..41bb136 100644
--- a/tpchgen/src/generators.rs
+++ b/tpchgen/src/generators.rs
@@ -23,16 +23,22 @@ pub struct NationGenerator<'a> {
 
 impl Default for NationGenerator<'_> {
     fn default() -> Self {
-        Self::new()
+        // arguments are ignored
+        Self::new(1.0, 1, 1)
     }
 }
 
 impl<'a> NationGenerator<'a> {
     /// Creates a new NationGenerator with default distributions and text pool
     ///
+    /// Nations does not depend on the scale factor or the part number. The signature of
+    /// this method is provided to be consistent with the other generators, but the
+    /// parameters are ignored. You can use [`NationGenerator::default`] to create a
+    /// default generator.
+    ///
     /// The generator's lifetime is `&'static` because it references global
     /// [`Distribution]`s and thus can be shared safely between threads.
-    pub fn new() -> NationGenerator<'static> {
+    pub fn new(_scale_factor: f64, _part: i32, _part_count: i32) -> NationGenerator<'static> {
         // Note: use explicit lifetime to ensure this remains `&'static`
         Self::new_with_distributions_and_text_pool(
             Distributions::static_default(),
@@ -208,16 +214,22 @@ pub struct RegionGenerator<'a> {
 
 impl Default for RegionGenerator<'_> {
     fn default() -> Self {
-        Self::new()
+        // arguments are ignored
+        Self::new(1.0, 1, 1)
     }
 }
 
 impl<'a> RegionGenerator<'a> {
     /// Creates a new RegionGenerator with default distributions and text pool
     ///
+    /// Regions does not depend on the scale factor or the part number. The signature of
+    /// this method is provided to be consistent with the other generators, but the
+    /// parameters are ignored. You can use [`RegionGenerator::default`] to create a
+    /// default generator.
+    ///
     /// Note the generator's lifetime is `&'static`. See [`NationGenerator`] for
     /// more details.
-    pub fn new() -> RegionGenerator<'static> {
+    pub fn new(_scale_factor: f64, _part: i32, _part_count: i32) -> RegionGenerator<'static> {
         // Note: use explicit lifetime to ensure this remains `&'static`
         Self::new_with_distributions_and_text_pool(
             Distributions::static_default(),
@@ -433,6 +445,11 @@ impl<'a> PartGenerator<'a> {
         }
     }
 
+    /// Return the row count for the given scale factor and generator part count
+    pub fn calculate_row_count(scale_factor: f64, part: i32, part_count: i32) -> i64 {
+        GenerateUtils::calculate_row_count(Self::SCALE_BASE, scale_factor, part, part_count)
+    }
+
     /// Returns an iterator over the part rows
     pub fn iter(&self) -> PartGeneratorIterator<'a> {
         PartGeneratorIterator::new(
@@ -444,12 +461,7 @@ impl<'a> PartGenerator<'a> {
                 self.part,
                 self.part_count,
             ),
-            GenerateUtils::calculate_row_count(
-                Self::SCALE_BASE,
-                self.scale_factor,
-                self.part,
-                self.part_count,
-            ),
+            Self::calculate_row_count(self.scale_factor, self.part, self.part_count),
         )
     }
 }
@@ -708,6 +720,11 @@ impl<'a> SupplierGenerator<'a> {
         }
     }
 
+    /// Return the row count for the given scale factor and generator part count
+    pub fn calculate_row_count(scale_factor: f64, part: i32, part_count: i32) -> i64 {
+        GenerateUtils::calculate_row_count(Self::SCALE_BASE, scale_factor, part, part_count)
+    }
+
     /// Returns an iterator over the supplier rows
     pub fn iter(&self) -> SupplierGeneratorIterator<'a> {
         SupplierGeneratorIterator::new(
@@ -719,12 +736,7 @@ impl<'a> SupplierGenerator<'a> {
                 self.part,
                 self.part_count,
             ),
-            GenerateUtils::calculate_row_count(
-                Self::SCALE_BASE,
-                self.scale_factor,
-                self.part,
-                self.part_count,
-            ),
+            Self::calculate_row_count(self.scale_factor, self.part, self.part_count),
         )
     }
 }
@@ -1010,6 +1022,11 @@ impl<'a> CustomerGenerator<'a> {
         }
     }
 
+    /// Return the row count for the given scale factor and generator part count
+    pub fn calculate_row_count(scale_factor: f64, part: i32, part_count: i32) -> i64 {
+        GenerateUtils::calculate_row_count(Self::SCALE_BASE, scale_factor, part, part_count)
+    }
+
     /// Returns an iterator over the customer rows
     pub fn iter(&self) -> CustomerGeneratorIterator<'a> {
         CustomerGeneratorIterator::new(
@@ -1021,12 +1038,7 @@ impl<'a> CustomerGenerator<'a> {
                 self.part,
                 self.part_count,
             ),
-            GenerateUtils::calculate_row_count(
-                Self::SCALE_BASE,
-                self.scale_factor,
-                self.part,
-                self.part_count,
-            ),
+            Self::calculate_row_count(self.scale_factor, self.part, self.part_count),
         )
     }
 }
@@ -1223,9 +1235,19 @@ impl<'a> PartSupplierGenerator<'a> {
         }
     }
 
+    /// Return the row count for the given scale factor and generator part count
+    pub fn calculate_row_count(scale_factor: f64, part: i32, part_count: i32) -> i64 {
+        // Use the part generator's scale base for start/row calculation
+        GenerateUtils::calculate_row_count(
+            PartGenerator::SCALE_BASE,
+            scale_factor,
+            part,
+            part_count,
+        )
+    }
+
     /// Returns an iterator over the part supplier rows
     pub fn iter(&self) -> PartSupplierGeneratorIterator<'a> {
-        // Use the part generator's scale base for start/row calculation
         let scale_base = PartGenerator::SCALE_BASE;
 
         PartSupplierGeneratorIterator::new(
@@ -1237,12 +1259,7 @@ impl<'a> PartSupplierGenerator<'a> {
                 self.part,
                 self.part_count,
             ),
-            GenerateUtils::calculate_row_count(
-                scale_base,
-                self.scale_factor,
-                self.part,
-                self.part_count,
-            ),
+            Self::calculate_row_count(self.scale_factor, self.part, self.part_count),
         )
     }
 }
@@ -1496,6 +1513,11 @@ impl<'a> OrderGenerator<'a> {
         }
     }
 
+    /// Return the row count for the given scale factor and generator part count
+    pub fn calculate_row_count(scale_factor: f64, part: i32, part_count: i32) -> i64 {
+        GenerateUtils::calculate_row_count(Self::SCALE_BASE, scale_factor, part, part_count)
+    }
+
     /// Returns an iterator over the order rows
     pub fn iter(&self) -> OrderGeneratorIterator<'a> {
         OrderGeneratorIterator::new(
@@ -1508,12 +1530,7 @@ impl<'a> OrderGenerator<'a> {
                 self.part,
                 self.part_count,
             ),
-            GenerateUtils::calculate_row_count(
-                Self::SCALE_BASE,
-                self.scale_factor,
-                self.part,
-                self.part_count,
-            ),
+            Self::calculate_row_count(self.scale_factor, self.part, self.part_count),
         )
     }
 
@@ -2211,7 +2228,7 @@ mod tests {
 
     #[test]
     fn test_nation_generator() {
-        let generator = NationGenerator::new();
+        let generator = NationGenerator::default();
         let nations: Vec<_> = generator.iter().collect();
 
         // TPC-H typically has 25 nations
@@ -2220,7 +2237,7 @@ mod tests {
 
     #[test]
     fn test_region_generator() {
-        let generator = RegionGenerator::new();
+        let generator = RegionGenerator::default();
         let regions: Vec<_> = generator.iter().collect();
 
         // TPC-H typically has 5 regions
@@ -2472,8 +2489,8 @@ mod tests {
         // Lifetimes of iterators should be independent of the generator that
         // created it. This test case won't compile if that's not the case.
 
-        let _iter: NationGeneratorIterator<'static> = NationGenerator::new().iter();
-        let _iter: RegionGeneratorIterator<'static> = RegionGenerator::new().iter();
+        let _iter: NationGeneratorIterator<'static> = NationGenerator::default().iter();
+        let _iter: RegionGeneratorIterator<'static> = RegionGenerator::default().iter();
         let _iter: PartGeneratorIterator<'static> = PartGenerator::new(0.1, 1, 1).iter();
         let _iter: SupplierGeneratorIterator<'static> = SupplierGenerator::new(0.1, 1, 1).iter();
         let _iter: CustomerGeneratorIterator<'static> = CustomerGenerator::new(0.1, 1, 1).iter();
diff --git a/tpchgen/tests/integration_tests.rs b/tpchgen/tests/integration_tests.rs
index 2b208c0..f5c53fe 100644
--- a/tpchgen/tests/integration_tests.rs
+++ b/tpchgen/tests/integration_tests.rs
@@ -56,7 +56,7 @@ where
 #[test]
 fn test_nation_sf_0_001() {
     let _sf = 0.001;
-    let generator = NationGenerator::new();
+    let generator = NationGenerator::default();
     test_generator(generator.iter(), "data/sf-0.001/nation.tbl.gz", |nation| {
         nation.to_string()
     });
@@ -65,7 +65,7 @@ fn test_nation_sf_0_001() {
 #[test]
 fn test_region_sf_0_001() {
     let _sf = 0.001;
-    let generator = RegionGenerator::new();
+    let generator = RegionGenerator::default();
     test_generator(generator.iter(), "data/sf-0.001/region.tbl.gz", |region| {
         region.to_string()
     });
@@ -131,8 +131,8 @@ fn test_lineitem_sf_0_001() {
 
 #[test]
 fn test_nation_sf_0_01() {
-    let _sf = 0.01;
-    let generator = NationGenerator::new();
+    let sf = 0.01;
+    let generator = NationGenerator::new(sf, 1, 1);
     test_generator(generator.iter(), "data/sf-0.01/nation.tbl.gz", |nation| {
         nation.to_string()
     });
@@ -140,8 +140,8 @@ fn test_nation_sf_0_01() {
 
 #[test]
 fn test_region_sf_0_01() {
-    let _sf = 0.01;
-    let generator = RegionGenerator::new();
+    let sf = 0.01;
+    let generator = RegionGenerator::new(sf, 1, 1);
     test_generator(generator.iter(), "data/sf-0.01/region.tbl.gz", |region| {
         region.to_string()
     });


Reviews:
user: login: alamb
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-22T11:56:32Z
user: login: clflushopt
body: LGTM; awesome stuff !!!
state: APPROVED
author_association: OWNER
submitted_at: 2025-03-25T00:34:40Z
user: login: alamb
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-25T11:24:50Z
user: login: alamb
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-25T11:24:54Z

Review comments:
diff_hunk: @@ -0,0 +1,128 @@
+//! Parallel data generation: [`Source`] and [`Sink`] and [`generate_in_chunks`]
path: tpchgen-cli/src/generate.rs
user: login: alamb
body: All the parallel generation code is in this module
created_at: 2025-03-22T11:53:10Z
author_association: COLLABORATOR
diff_hunk: @@ -10,22 +10,42 @@
 //! -T, --tables     Tables to generate data for
 //! -F, --format     Output format for the data (CSV or Parquet)
 //! -O, --output     Output directory for the generated data
-//
-// The main function is the entry point for the CLI and it uses the `clap` crate
-// to parse the command line arguments and then generate the data.
-
-// tpchgen-cli/src/main.rs
+//! -v, --verbose    Verbose output
path: tpchgen-cli/src/main.rs
user: login: alamb
body: new logging feature
created_at: 2025-03-22T11:54:09Z
author_association: COLLABORATOR
diff_hunk: @@ -90,14 +120,53 @@ enum OutputFormat {
     Parquet,
 }
 
-fn main() -> io::Result<()> {
+#[tokio::main]
+async fn main() -> io::Result<()> {
     // Parse command line arguments
     let cli = Cli::parse();
-    cli.main()
+    cli.main().await
+}
+
+/// macro to create a Cli function for generating a table
path: tpchgen-cli/src/main.rs
user: login: alamb
body: avoided a lot of repetition with macros
created_at: 2025-03-22T11:54:28Z
author_association: COLLABORATOR
diff_hunk: @@ -23,16 +23,21 @@ pub struct NationGenerator<'a> {
 
 impl Default for NationGenerator<'_> {
     fn default() -> Self {
-        Self::new()
+        // arguments are ignored
+        Self::new(1.0, 1, 1)
     }
 }
 
 impl<'a> NationGenerator<'a> {
     /// Creates a new NationGenerator with default distributions and text pool
     ///
+    /// Nations does not depend on the scale factor or the part number. The signature of
+    /// this method is provided to be consistent with the other generators, but the
+    /// parameters are ignored.
+    ///
     /// The generator's lifetime is `&'static` because it references global
     /// [`Distribution]`s and thus can be shared safely between threads.
-    pub fn new() -> NationGenerator<'static> {
+    pub fn new(_scale_factor: f64, _part: i32, _part_count: i32) -> NationGenerator<'static> {
path: tpchgen/src/generators.rs
user: login: alamb
body: I made these methods take the same arguments as the other generators to simplify the macros (all generators now have the same signature)
created_at: 2025-03-22T11:55:21Z
author_association: COLLABORATOR
diff_hunk: @@ -433,6 +443,11 @@ impl<'a> PartGenerator<'a> {
         }
     }
 
+    /// Return the row count for the given scale factor and generator part count
path: tpchgen/src/generators.rs
user: login: alamb
body: I refactored out the row count calculation into a function that can be called from outside (needed to calculate how many parts for each table)
created_at: 2025-03-22T11:56:27Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,134 @@
+//! Parallel data generation: [`Source`] and [`Sink`] and [`generate_in_chunks`]
+//!
+//! These traits and function are used to generate data in parallel and write it to a sink
+//! in streaming fashion (chunks). This is useful for generating large datasets that don't fit in memory.
+
+use futures::StreamExt;
+use log::debug;
+use std::collections::VecDeque;
+use std::io;
+use std::sync::{Arc, Mutex};
+use tokio::task::JoinSet;
+
+/// Something that knows how to generate data into a buffer
+///
+/// For example, this is implemented for the different generators in the tpchgen
+/// crate
+pub trait Source: Send {
+    /// generates the data for this generator into the buffer, returning the buffer.
+    fn create(self, buffer: Vec<u8>) -> Vec<u8>;
+}
+
+/// Something that can write the contents of a buffer somewhere
+///
+/// For example, this is implemented for a file writer
+pub trait Sink: Send {
+    /// Write all data from the buffer to the sink
+    fn sink(&mut self, buffer: &[u8]) -> Result<(), io::Error>;
+
+    fn finish(self) -> Result<(), io::Error>;
path: tpchgen-cli/src/generate.rs
user: login: clflushopt
body: question: I have no preference here, but I was wondering if we shouldn't rename `finish` to `flush` to keep it "semantically closed" to how a sink might behave e.g. `flush()` calls `flush` and closes the file or buffered writer underlying.
created_at: 2025-03-25T00:28:07Z
author_association: OWNER
diff_hunk: @@ -0,0 +1,134 @@
+//! Parallel data generation: [`Source`] and [`Sink`] and [`generate_in_chunks`]
+//!
+//! These traits and function are used to generate data in parallel and write it to a sink
+//! in streaming fashion (chunks). This is useful for generating large datasets that don't fit in memory.
+
+use futures::StreamExt;
+use log::debug;
+use std::collections::VecDeque;
+use std::io;
+use std::sync::{Arc, Mutex};
+use tokio::task::JoinSet;
+
+/// Something that knows how to generate data into a buffer
+///
+/// For example, this is implemented for the different generators in the tpchgen
+/// crate
+pub trait Source: Send {
+    /// generates the data for this generator into the buffer, returning the buffer.
+    fn create(self, buffer: Vec<u8>) -> Vec<u8>;
+}
+
+/// Something that can write the contents of a buffer somewhere
+///
+/// For example, this is implemented for a file writer
+pub trait Sink: Send {
+    /// Write all data from the buffer to the sink
+    fn sink(&mut self, buffer: &[u8]) -> Result<(), io::Error>;
+
+    fn finish(self) -> Result<(), io::Error>;
+}
+
+/// Creates data from a set of [`Source`] in parallel sending data in order to the Sink.
path: tpchgen-cli/src/generate.rs
user: login: clflushopt
body: suggestion: Can you clarify that `Source` here is a distribution or row generator and clarify that order means  order in which generators are specified.
created_at: 2025-03-25T00:30:01Z
author_association: OWNER
diff_hunk: @@ -0,0 +1,134 @@
+//! Parallel data generation: [`Source`] and [`Sink`] and [`generate_in_chunks`]
+//!
+//! These traits and function are used to generate data in parallel and write it to a sink
+//! in streaming fashion (chunks). This is useful for generating large datasets that don't fit in memory.
+
+use futures::StreamExt;
+use log::debug;
+use std::collections::VecDeque;
+use std::io;
+use std::sync::{Arc, Mutex};
+use tokio::task::JoinSet;
+
+/// Something that knows how to generate data into a buffer
+///
+/// For example, this is implemented for the different generators in the tpchgen
+/// crate
+pub trait Source: Send {
+    /// generates the data for this generator into the buffer, returning the buffer.
+    fn create(self, buffer: Vec<u8>) -> Vec<u8>;
+}
+
+/// Something that can write the contents of a buffer somewhere
+///
+/// For example, this is implemented for a file writer
+pub trait Sink: Send {
+    /// Write all data from the buffer to the sink
+    fn sink(&mut self, buffer: &[u8]) -> Result<(), io::Error>;
+
+    fn finish(self) -> Result<(), io::Error>;
+}
+
+/// Creates data from a set of [`Source`] in parallel sending data in order to the Sink.
+///
+///
+/// G: Generator
+/// I: Iterator<Item = G>
+/// S: Sink that writes buffers somewhere
+pub async fn generate_in_chunks<G, I, S>(mut sink: S, generators: I) -> Result<(), io::Error>
+where
+    G: Source + 'static,
+    I: Iterator<Item = G>,
+    S: Sink + 'static,
+{
+    let recycler = BufferRecycler::new();
+
+    // use all cores to make data
+    let num_tasks = num_cpus::get();
+    debug!("Generating with {num_tasks} parallel tasks");
+
+    // create a channel to communicate between the generator tasks and the writer task
+    let (tx, mut rx) = tokio::sync::mpsc::channel(num_tasks);
+
+    let generators_and_recyclers = generators.map(|generator| (generator, recycler.clone()));
+
+    // convert to an async stream to run on tokio
+    let mut stream = futures::stream::iter(generators_and_recyclers)
+        // each generator writes to a buffer
+        .map(async |(generator, recycler)| {
+            let buffer = recycler.new_buffer(1024 * 1024 * 8);
+            // do the work in a task (on a different thread)
+            let mut join_set = JoinSet::new();
+            join_set.spawn(async move { generator.create(buffer) });
+            // wait for the task to be done and return the result
+            join_set
+                .join_next()
+                .await
+                .expect("had one item")
+                .expect("join_next join is infallible unless task panics")
+        })
+        // run in parallel
+        .buffered(num_tasks)
+        .map(async |buffer| {
+            // send the buffer to the writer task, in order.
+
+            // ignoring error (if the writer errored it means the channel is closed / the program is exiting)
+            if let Err(e) = tx.send(buffer).await {
+                debug!("Error sending buffer to writer: {e}");
+            }
+        });
+
+    let captured_recycler = recycler.clone();
+    let writer_task = tokio::task::spawn_blocking(move || {
+        while let Some(buffer) = rx.blocking_recv() {
+            sink.sink(&buffer)?;
+            captured_recycler.return_buffer(buffer);
+        }
+        sink.finish()
+    });
+
+    // drive the stream to completion
+    while let Some(write_task) = stream.next().await {
+        write_task.await; // sends the buffer to the writer task
+    }
+    drop(stream); // drop any stream references
+    drop(tx); // drop last tx reference to stop the writer
+
+    // wait for writer to finish
+    debug!("waiting for writer task to complete");
+    writer_task.await.expect("writer task panicked")
+}
+
+/// A simple buffer recycler to avoid allocating new buffers for each part
+///
+/// Clones share the same undrlying recycler, so it is not thread safe
path: tpchgen-cli/src/generate.rs
user: login: clflushopt
body: ```suggestion
/// Clones share the same underlying recycler, so it is not thread safe
```
created_at: 2025-03-25T00:32:14Z
author_association: OWNER
diff_hunk: @@ -0,0 +1,134 @@
+//! Parallel data generation: [`Source`] and [`Sink`] and [`generate_in_chunks`]
+//!
+//! These traits and function are used to generate data in parallel and write it to a sink
+//! in streaming fashion (chunks). This is useful for generating large datasets that don't fit in memory.
+
+use futures::StreamExt;
+use log::debug;
+use std::collections::VecDeque;
+use std::io;
+use std::sync::{Arc, Mutex};
+use tokio::task::JoinSet;
+
+/// Something that knows how to generate data into a buffer
+///
+/// For example, this is implemented for the different generators in the tpchgen
+/// crate
+pub trait Source: Send {
+    /// generates the data for this generator into the buffer, returning the buffer.
+    fn create(self, buffer: Vec<u8>) -> Vec<u8>;
+}
+
+/// Something that can write the contents of a buffer somewhere
+///
+/// For example, this is implemented for a file writer
+pub trait Sink: Send {
+    /// Write all data from the buffer to the sink
+    fn sink(&mut self, buffer: &[u8]) -> Result<(), io::Error>;
+
+    fn finish(self) -> Result<(), io::Error>;
path: tpchgen-cli/src/generate.rs
user: login: alamb
body: Excellent idea -- done in 4985895
created_at: 2025-03-25T11:24:50Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,134 @@
+//! Parallel data generation: [`Source`] and [`Sink`] and [`generate_in_chunks`]
+//!
+//! These traits and function are used to generate data in parallel and write it to a sink
+//! in streaming fashion (chunks). This is useful for generating large datasets that don't fit in memory.
+
+use futures::StreamExt;
+use log::debug;
+use std::collections::VecDeque;
+use std::io;
+use std::sync::{Arc, Mutex};
+use tokio::task::JoinSet;
+
+/// Something that knows how to generate data into a buffer
+///
+/// For example, this is implemented for the different generators in the tpchgen
+/// crate
+pub trait Source: Send {
+    /// generates the data for this generator into the buffer, returning the buffer.
+    fn create(self, buffer: Vec<u8>) -> Vec<u8>;
+}
+
+/// Something that can write the contents of a buffer somewhere
+///
+/// For example, this is implemented for a file writer
+pub trait Sink: Send {
+    /// Write all data from the buffer to the sink
+    fn sink(&mut self, buffer: &[u8]) -> Result<(), io::Error>;
+
+    fn finish(self) -> Result<(), io::Error>;
+}
+
+/// Creates data from a set of [`Source`] in parallel sending data in order to the Sink.
path: tpchgen-cli/src/generate.rs
user: login: alamb
body: Done in b611865
created_at: 2025-03-25T11:24:54Z
author_association: COLLABORATOR

PR comments:
user: login: alamb
created_at: 2025-03-21T15:46:48Z
body: Ok, I need to stop for today, but this is looking promising
author_association: COLLABORATOR
user: login: alamb
created_at: 2025-03-24T13:23:56Z
body: I plan to merge this PR tomorrow unless anyone else would like time to comment on it. 
author_association: COLLABORATOR
user: login: clflushopt
created_at: 2025-03-24T23:32:42Z
body: I will take a look at it tonight ! Thanks @alamb 
author_association: OWNER
user: login: alamb
created_at: 2025-03-25T11:30:46Z
body: BTW with the other improvements on main, when merging it takes less than 5 seconds on my laptop to make TPCH SF 10 🚀 

```shell
time target/release/tpchgen-cli -s 10  --output-dir=/tmp/tpchdbgen-rs

real	0m4.895s
user	0m36.096s
sys	0m2.910s
```
author_association: COLLABORATOR