Repository: y-scope/clp
Pull request #831: refactor(clp-py-utils): Move CLP metadata DB table-creation logic into utility module; Conditionally create the datasets table when using the `CLP_S` storage engine.
Author: login: Bill-hbrhbr
Created: 2025-04-17T07:41:52Z | Merged: 2025-04-23T20:00:39Z

<!-- markdownlint-disable MD012 -->

<!--
Set the PR title to a meaningful commit message that:

* is in imperative form.
* follows the Conventional Commits specification (https://www.conventionalcommits.org).
  * See https://github.com/commitizen/conventional-commit-types/blob/master/index.json for possible
    types.

Example:

fix: Don't add implicit wildcards ('*') at the beginning and the end of a query (fixes #390).
-->

# Description

<!-- Describe what this request will change/fix and provide any details necessary for reviewers. -->

* Add a function to create the `clp_datasets` table.  
* Move creation logic for various tables into a reusable Python utility.  
* Add a helper function to create the set of following tables with an optional `dataset` specification that defaults to `None`:
  * `clp_{dataset}_archives`  
  * `clp_{dataset}_tags`  
  * `clp_{dataset}_archive_tags`  
  * `clp_{dataset}_files`  
    * If `dataset` is specified, also create `clp_{dataset}_column_metadata`  
  * If not, create the default versions of the table (used by clp):  
    * `clp_archives`, `clp_tags`, `clp_archive_tags`, `clp_files`
* Pass `--storage-engine` flag from `start-clp.py` all the way to `initialize-clp-metadata-db.py` and create `clp_datasets` if the storage engine is `clp-s`

This PR succeeds #819, #839 and covers the following parts of the dataset feature implementation [plan](https://docs.google.com/document/d/1BXDG1LSFOS7I52BRc0s1ahk_9MebCiH2HlLefLJ9unM/edit?tab=t.0#heading=h.kjcfz55f6si7):
* The part of Step 2 regarding creating the `clp_datasets` table.
* Will also help Step 3 when we create new tables upon seeing a new `dataset`

# Checklist

<!-- Ensure each item below is satisfied and indicate so by inserting an `x` within each `[ ]`. -->

* [X] The PR satisfies the [contribution guidelines][yscope-contrib-guidelines].
* [X] This is a breaking change and that has been indicated in the PR title, OR this isn't a
  breaking change.
* [X] Necessary docs have been updated, OR no docs need to be updated.

# Validation performed

<!-- Describe what tests and validation you performed on the change. -->

* [X] Tested with clp-package. Tables are created successfully as usual. With `CLP_S`, the `clp_datasets` table is also created.

For `CLP_S`:
![image](https://github.com/user-attachments/assets/5b62d246-0278-4a73-9970-98b52a48e4cd)




[yscope-contrib-guidelines]: https://docs.yscope.com/dev-guide/contrib-guides-overview.html


<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

## Summary by CodeRabbit

- **New Features**
	- Introduced new utilities for creating and managing metadata tables in MariaDB or MySQL databases.
- **Refactor**
	- Streamlined the metadata database initialization process by consolidating multiple table creation steps into a single function call.
- **Chores**
	- Added new string constants for table suffixes to improve table naming consistency.
	- Enhanced command-line interfaces to require and propagate storage engine configuration for database setup.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Linked issue #390: clp-s: Implicit wildcard behaviour is unintuitive
### Request

For clp-s the way that we treat wildcards is unintuitive.

The rules are that if a user adds any explicit wildcards we don't add any implicit ones, and we only add implicit wildcards at the start and end if the predicate contains a space.

To make things simpler for users we should make the user use explicit wildcards for everything.

### Possible implementation

Instead of conditionally adding wildcards when processing a query we will simply not add implicit wildcards.

Diff:
diff --git a/components/clp-package-utils/clp_package_utils/scripts/start_clp.py b/components/clp-package-utils/clp_package_utils/scripts/start_clp.py
index 525ddd54d8..660458744b 100755
--- a/components/clp-package-utils/clp_package_utils/scripts/start_clp.py
+++ b/components/clp-package-utils/clp_package_utils/scripts/start_clp.py
@@ -231,6 +231,7 @@ def create_db_tables(
         "python3",
         str(clp_py_utils_dir / "create-db-tables.py"),
         "--config", str(container_clp_config.logs_directory / db_config_filename),
+        "--storage-engine", str(container_clp_config.package.storage_engine),
     ]
     # fmt: on
 
diff --git a/components/clp-py-utils/clp_py_utils/clp_config.py b/components/clp-py-utils/clp_py_utils/clp_config.py
index f091e7a1ff..5c9fc9778d 100644
--- a/components/clp-py-utils/clp_py_utils/clp_config.py
+++ b/components/clp-py-utils/clp_py_utils/clp_config.py
@@ -36,7 +36,13 @@
 QUERY_TASKS_TABLE_NAME = "query_tasks"
 COMPRESSION_JOBS_TABLE_NAME = "compression_jobs"
 COMPRESSION_TASKS_TABLE_NAME = "compression_tasks"
+
+ARCHIVE_TAGS_TABLE_SUFFIX = "archive_tags"
 ARCHIVES_TABLE_SUFFIX = "archives"
+COLUMN_METADATA_TABLE_SUFFIX = "column_metadata"
+DATASETS_TABLE_SUFFIX = "datasets"
+FILES_TABLE_SUFFIX = "files"
+TAGS_TABLE_SUFFIX = "tags"
 
 OS_RELEASE_FILE_PATH = pathlib.Path("etc") / "os-release"
 
diff --git a/components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py b/components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py
new file mode 100644
index 0000000000..f5420aaf16
--- /dev/null
+++ b/components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py
@@ -0,0 +1,142 @@
+from __future__ import annotations
+
+from clp_py_utils.clp_config import (
+    ARCHIVE_TAGS_TABLE_SUFFIX,
+    ARCHIVES_TABLE_SUFFIX,
+    CLP_DEFAULT_DATASET_NAME,
+    COLUMN_METADATA_TABLE_SUFFIX,
+    DATASETS_TABLE_SUFFIX,
+    FILES_TABLE_SUFFIX,
+    TAGS_TABLE_SUFFIX,
+)
+
+
+def _create_archives_table(db_cursor, archives_table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{archives_table_name}` (
+            `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
+            `id` VARCHAR(64) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `uncompressed_size` BIGINT NOT NULL,
+            `size` BIGINT NOT NULL,
+            `creator_id` VARCHAR(64) NOT NULL,
+            `creation_ix` INT NOT NULL,
+            KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
+            UNIQUE KEY `archive_id` (`id`) USING BTREE,
+            PRIMARY KEY (`pagination_id`)
+        )
+        """
+    )
+
+
+def _create_tags_table(db_cursor, tags_table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{tags_table_name}` (
+            `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
+            `tag_name` VARCHAR(255) NOT NULL,
+            UNIQUE KEY (`tag_name`) USING BTREE,
+            PRIMARY KEY (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_archive_tags_table(
+    db_cursor, archive_tags_table_name: str, archives_table_name: str, tags_table_name: str
+) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{archive_tags_table_name}` (
+            `archive_id` VARCHAR(64) NOT NULL,
+            `tag_id` INT unsigned NOT NULL,
+            PRIMARY KEY (`archive_id`,`tag_id`),
+            FOREIGN KEY (`archive_id`) REFERENCES `{archives_table_name}` (`id`),
+            FOREIGN KEY (`tag_id`) REFERENCES `{tags_table_name}` (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_files_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{FILES_TABLE_SUFFIX}` (
+            `id` VARCHAR(64) NOT NULL,
+            `orig_file_id` VARCHAR(64) NOT NULL,
+            `path` VARCHAR(12288) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `num_uncompressed_bytes` BIGINT NOT NULL,
+            `begin_message_ix` BIGINT NOT NULL,
+            `num_messages` BIGINT NOT NULL,
+            `archive_id` VARCHAR(64) NOT NULL,
+            KEY `files_path` (path(768)) USING BTREE,
+            KEY `files_archive_id` (`archive_id`) USING BTREE,
+            PRIMARY KEY (`id`)
+        ) ROW_FORMAT=DYNAMIC
+        """
+    )
+
+
+def _create_column_metadata_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `name` VARCHAR(512) NOT NULL,
+            `type` TINYINT NOT NULL,
+            PRIMARY KEY (`name`, `type`)
+        )
+        """
+    )
+
+
+def create_datasets_table(db_cursor, table_prefix: str) -> None:
+    """
+    Creates the dataset information table.
+
+    :param db_cursor: The database cursor to execute the table creation.
+    :param table_prefix: A string to prepend to the table name.
+    """
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{DATASETS_TABLE_SUFFIX}` (
+            `name` VARCHAR(255) NOT NULL,
+            `archive_storage_directory` VARCHAR(4096) NOT NULL,
+            PRIMARY KEY (`name`)
+        )
+        """
+    )
+
+
+def create_metadata_db_tables(db_cursor, table_prefix: str, dataset: str | None = None) -> None:
+    """
+    Creates the standard set of tables for CLP's metadata.
+
+    :param db_cursor: The database cursor to execute the table creations.
+    :param table_prefix: A string to prepend to all table names.
+    :param dataset: If set, all tables will be named in a dataset-specific manner.
+    """
+    if dataset is not None:
+        table_prefix = f"{table_prefix}{dataset}_"
+
+    archives_table_name = f"{table_prefix}{ARCHIVES_TABLE_SUFFIX}"
+    tags_table_name = f"{table_prefix}{TAGS_TABLE_SUFFIX}"
+    archive_tags_table_name = f"{table_prefix}{ARCHIVE_TAGS_TABLE_SUFFIX}"
+
+    # TODO: Update this to
+    # {table_prefix}{CLP_DEFAULT_DATASET_NAME}_{COLUMN_METADATA_TABLE_SUFFIX} when we can also
+    # change the indexer to match.
+    column_metadata_table_name = (
+        f"{table_prefix}{COLUMN_METADATA_TABLE_SUFFIX}_{CLP_DEFAULT_DATASET_NAME}"
+    )
+
+    _create_archives_table(db_cursor, archives_table_name)
+    _create_tags_table(db_cursor, tags_table_name)
+    _create_archive_tags_table(
+        db_cursor, archive_tags_table_name, archives_table_name, tags_table_name
+    )
+    _create_files_table(db_cursor, table_prefix)
+    _create_column_metadata_table(db_cursor, column_metadata_table_name)
diff --git a/components/clp-py-utils/clp_py_utils/create-db-tables.py b/components/clp-py-utils/clp_py_utils/create-db-tables.py
index b376fdf640..1fb3f07c6f 100644
--- a/components/clp-py-utils/clp_py_utils/create-db-tables.py
+++ b/components/clp-py-utils/clp_py_utils/create-db-tables.py
@@ -4,6 +4,8 @@
 import subprocess
 import sys
 
+from clp_py_utils.clp_config import StorageEngine
+
 # Setup logging
 # Create logger
 logger = logging.getLogger(__file__)
@@ -18,9 +20,17 @@
 def main(argv):
     args_parser = argparse.ArgumentParser(description="Creates database tables for CLP.")
     args_parser.add_argument("--config", required=True, help="Database config file.")
+    args_parser.add_argument(
+        "--storage-engine",
+        type=str,
+        choices=[engine.value for engine in StorageEngine],
+        required=True,
+        help="Compression storage engine to use.",
+    )
     parsed_args = args_parser.parse_args(argv[1:])
 
     config_file_path = pathlib.Path(parsed_args.config)
+    storage_engine = StorageEngine(parsed_args.storage_engine)
 
     script_dir = pathlib.Path(__file__).parent.resolve()
 
@@ -28,6 +38,7 @@ def main(argv):
     cmd = [
         "python3", str(script_dir / "initialize-clp-metadata-db.py"),
         "--config", str(config_file_path),
+        "--storage-engine", str(storage_engine),
     ]
     # fmt: on
     subprocess.run(cmd, check=True)
diff --git a/components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py b/components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py
index 83eef63fc4..0dc547964e 100644
--- a/components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py
+++ b/components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py
@@ -3,10 +3,18 @@
 import logging
 import sys
 from contextlib import closing
+from pathlib import Path
 
 from sql_adapter import SQL_Adapter
 
-from clp_py_utils.clp_config import Database
+from clp_py_utils.clp_config import (
+    Database,
+    StorageEngine,
+)
+from clp_py_utils.clp_metadata_db_utils import (
+    create_datasets_table,
+    create_metadata_db_tables,
+)
 from clp_py_utils.core import read_yaml_config_file
 
 # Setup logging
@@ -23,88 +31,34 @@
 def main(argv):
     args_parser = argparse.ArgumentParser(description="Sets up CLP's metadata tables.")
     args_parser.add_argument("--config", required=True, help="Database config file.")
+    args_parser.add_argument(
+        "--storage-engine",
+        type=str,
+        choices=[engine.value for engine in StorageEngine],
+        required=True,
+        help="Storage engine to create tables for.",
+    )
     parsed_args = args_parser.parse_args(argv[1:])
 
+    config_file_path = Path(parsed_args.config)
+    storage_engine = StorageEngine(parsed_args.storage_engine)
+
     try:
-        database_config = Database.parse_obj(read_yaml_config_file(parsed_args.config))
+        database_config = Database.parse_obj(read_yaml_config_file(config_file_path))
         if database_config is None:
-            raise ValueError(f"Database configuration file '{parsed_args.config}' is empty.")
+            raise ValueError(f"Database configuration file '{config_file_path}' is empty.")
         sql_adapter = SQL_Adapter(database_config)
         clp_db_connection_params = database_config.get_clp_connection_params_and_type(True)
         table_prefix = clp_db_connection_params["table_prefix"]
         with closing(sql_adapter.create_connection(True)) as metadata_db, closing(
             metadata_db.cursor(dictionary=True)
         ) as metadata_db_cursor:
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}archives` (
-                    `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
-                    `id` VARCHAR(64) NOT NULL,
-                    `begin_timestamp` BIGINT NOT NULL,
-                    `end_timestamp` BIGINT NOT NULL,
-                    `uncompressed_size` BIGINT NOT NULL,
-                    `size` BIGINT NOT NULL,
-                    `creator_id` VARCHAR(64) NOT NULL,
-                    `creation_ix` INT NOT NULL,
-                    KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
-                    UNIQUE KEY `archive_id` (`id`) USING BTREE,
-                    PRIMARY KEY (`pagination_id`)
-                )
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}tags` (
-                    `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
-                    `tag_name` VARCHAR(255) NOT NULL,
-                    UNIQUE KEY (`tag_name`) USING BTREE,
-                    PRIMARY KEY (`tag_id`)
-                )
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}archive_tags` (
-                    `archive_id` VARCHAR(64) NOT NULL,
-                    `tag_id` INT unsigned NOT NULL,
-                    PRIMARY KEY (`archive_id`,`tag_id`),
-                    FOREIGN KEY (`archive_id`) REFERENCES `{table_prefix}archives` (`id`),
-                    FOREIGN KEY (`tag_id`) REFERENCES `{table_prefix}tags` (`tag_id`)
-                )
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}files` (
-                    `id` VARCHAR(64) NOT NULL,
-                    `orig_file_id` VARCHAR(64) NOT NULL,
-                    `path` VARCHAR(12288) NOT NULL,
-                    `begin_timestamp` BIGINT NOT NULL,
-                    `end_timestamp` BIGINT NOT NULL,
-                    `num_uncompressed_bytes` BIGINT NOT NULL,
-                    `begin_message_ix` BIGINT NOT NULL,
-                    `num_messages` BIGINT NOT NULL,
-                    `archive_id` VARCHAR(64) NOT NULL,
-                    KEY `files_path` (path(768)) USING BTREE,
-                    KEY `files_archive_id` (`archive_id`) USING BTREE,
-                    PRIMARY KEY (`id`)
-                ) ROW_FORMAT=DYNAMIC
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}column_metadata_default` (
-                    `name` VARCHAR(512) NOT NULL,
-                    `type` TINYINT NOT NULL,
-                    PRIMARY KEY (`name`, `type`)
-                )
-                """
-            )
-
+            # TODO: After the dataset feature is fully implemented, for clp-json:
+            # 1. Populate the datasets table with the name and path for the "default" dataset.
+            # 2. Change the metadata tables to be specific to the "default" dataset.
+            if StorageEngine.CLP_S == storage_engine:
+                create_datasets_table(metadata_db_cursor, table_prefix)
+            create_metadata_db_tables(metadata_db_cursor, table_prefix)
             metadata_db.commit()
     except:
         logger.exception("Failed to create clp metadata tables.")


Reviews:
user: login: coderabbitai[bot]
body: **Actionable comments posted: 2**

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between ffbd533a306dd9403e6a2226f723a359f4c2ca6b and 3f19234bd3ed9b72df80da9a41c27acd4aebee9a.

</details>

<details>
<summary>📒 Files selected for processing (7)</summary>

* `components/clp-package-utils/clp_package_utils/scripts/compress.py` (2 hunks)
* `components/clp-package-utils/clp_package_utils/scripts/native/compress.py` (3 hunks)
* `components/clp-py-utils/clp_py_utils/clp_config.py` (1 hunks)
* `components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py` (3 hunks)
* `components/clp-py-utils/clp_py_utils/sql_table_schema_utils.py` (1 hunks)
* `components/job-orchestration/job_orchestration/scheduler/compress/compression_scheduler.py` (5 hunks)
* `components/job-orchestration/job_orchestration/scheduler/job_config.py` (1 hunks)

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🧬 Code Graph Analysis (1)</summary>

<details>
<summary>components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py (3)</summary><blockquote>

<details>
<summary>components/clp-py-utils/clp_py_utils/clp_config.py (1)</summary>

* `Database` (76-155)

</details>
<details>
<summary>components/clp-py-utils/clp_py_utils/core.py (1)</summary>

* `read_yaml_config_file` (38-44)

</details>
<details>
<summary>components/clp-py-utils/clp_py_utils/sql_table_schema_utils.py (2)</summary>

* `create_archives_table` (1-18)
* `create_column_metadata_table` (21-30)

</details>

</blockquote></details>

</details>

</details>

<details>
<summary>🔇 Additional comments (17)</summary><blockquote>

<details>
<summary>components/clp-py-utils/clp_py_utils/clp_config.py (1)</summary>

`40-41`: **Constants for dataset-related table suffixes look good**

The addition of these constants follows the established pattern for table naming and will provide consistency when constructing dataset-specific table names throughout the codebase.

</details>
<details>
<summary>components/job-orchestration/job_orchestration/scheduler/job_config.py (2)</summary>

`28-28`: **Dataset field added with sensible default**

The addition of the dataset field with a default value of "default" maintains backward compatibility while enabling the new dataset feature.

---

`34-34`: **Dataset field added with sensible default**

The addition of the dataset field with a default value of "default" maintains backward compatibility while enabling the new dataset feature.

</details>
<details>
<summary>components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py (4)</summary>

`9-16`: **Updated imports for new constants and utilities**

The import statements have been updated to include the new constants and utility functions, which is appropriate.

---

`44-44`: **Good refactoring to use the utility function**

Replacing inline SQL with a call to the utility function improves maintainability and ensures consistency in table schemas.

---

`90-93`: **Simplified primary key for datasets table**

The primary key for the datasets table has been simplified to use only the `name` column, which is appropriate since dataset names should be unique.

---

`97-99`: **Good refactoring to use the utility function**

Using the `create_column_metadata_table` utility function instead of inline SQL improves maintainability and ensures consistency.

</details>
<details>
<summary>components/clp-py-utils/clp_py_utils/sql_table_schema_utils.py (2)</summary>

`1-18`: **Well-structured utility function for archives table creation**

This utility function encapsulates the SQL for creating an archives table, making it reusable across the codebase. The table structure includes all necessary fields and proper indexing.

---

`21-30`: **Well-structured utility function for column metadata table creation**

This utility function encapsulates the SQL for creating a column metadata table, which will be reused when creating dataset-specific metadata tables. The composite primary key on `name` and `type` is appropriate.

</details>
<details>
<summary>components/clp-package-utils/clp_package_utils/scripts/compress.py (2)</summary>

`85-87`: **Dataset flag handling looks good!**

The conditional check ensures the `--dataset` flag is only added when the dataset argument has a value, which is a good practice.

---

`151-153`: **Implementation of the `--dataset` parameter looks good!**

The default value "default" is appropriate for maintaining backward compatibility, and the help text clearly communicates the purpose of this parameter.

</details>
<details>
<summary>components/clp-package-utils/clp_package_utils/scripts/native/compress.py (3)</summary>

`143-143`: **Dataset parameter inclusion in FsInputConfig looks good!**

The parameter is properly passed to the input configuration object, enabling dataset-specific compression for filesystem inputs.

---

`156-156`: **Dataset parameter inclusion in S3InputConfig looks good!**

The parameter is properly passed to the input configuration object, enabling dataset-specific compression for S3 inputs.

---

`208-210`: **Implementation of the `--dataset` parameter looks good!**

The default value "default" matches the implementation in compress.py, maintaining consistency across the codebase.

</details>
<details>
<summary>components/job-orchestration/job_orchestration/scheduler/compress/compression_scheduler.py (3)</summary>

`156-158`: **Function parameter update looks good!**

The `search_and_schedule_new_tasks` function is properly updated to accept the storage engine parameter, which is needed for the dataset-specific logic.

---

`411-412`: **Storage engine extraction looks good!**

The storage engine is properly extracted from the configuration to be passed to the task scheduling function.

---

`418-418`: **Parameter passing looks good!**

The storage engine is correctly passed to the `search_and_schedule_new_tasks` function.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T09:55:26Z
user: login: Bill-hbrhbr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T16:54:34Z
user: login: Bill-hbrhbr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T16:54:53Z
user: login: coderabbitai[bot]
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T16:55:07Z
user: login: coderabbitai[bot]
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T16:55:25Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 4**

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 5ffcf92eb8eca44b7e1d3290a7d646b0b9cc8ac4 and 9d9d0d6df779eb304bdc8403e0c53a48e1dca867.

</details>

<details>
<summary>📒 Files selected for processing (3)</summary>

* `components/clp-py-utils/clp_py_utils/clp_config.py` (1 hunks)
* `components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py` (2 hunks)
* `components/clp-py-utils/clp_py_utils/sql_table_schema_utils.py` (1 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (1)</summary>

* components/clp-py-utils/clp_py_utils/clp_config.py

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🧬 Code Graph Analysis (1)</summary>

<details>
<summary>components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py (3)</summary><blockquote>

<details>
<summary>components/clp-py-utils/clp_py_utils/clp_config.py (1)</summary>

* `Database` (80-159)

</details>
<details>
<summary>components/clp-py-utils/clp_py_utils/core.py (1)</summary>

* `read_yaml_config_file` (38-44)

</details>
<details>
<summary>components/clp-py-utils/clp_py_utils/sql_table_schema_utils.py (4)</summary>

* `create_archives_table` (1-18)
* `create_tags_table` (21-31)
* `create_archive_tags_table` (34-45)
* `create_files_table` (48-66)

</details>

</blockquote></details>

</details>

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (3)</summary>

* GitHub Check: lint-check (ubuntu-latest)
* GitHub Check: lint-check (macos-latest)
* GitHub Check: build (macos-latest)

</details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T13:49:34Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 1**

<details>
<summary>🔭 Outside diff range comments (1)</summary><blockquote>

<details>
<summary>components/clp-py-utils/clp_py_utils/sql_table_schema_utils.py (1)</summary><blockquote>

`105-123`: _🛠️ Refactor suggestion_

**`create_metadata_db_tables` never records the dataset in `clp_datasets`**

The PR description emphasises the new `clp_datasets` registry table, yet this helper never invokes `create_datasets_table`.  Consequently, compression jobs that rely solely on this helper will create dataset‑specific tables without inserting the dataset into the global registry, leaving the schema in an inconsistent state.

Consider:

```diff
@@
     db_cursor, table_prefix: str, dataset: typing.Optional[str] = None
 ) -> None:
     if dataset is not None:
         table_prefix = f"{table_prefix}{dataset}_"
         create_column_metadata_table(db_cursor, f"{table_prefix}{COLUMN_METADATA_TABLE_SUFFIX}")
+        # Ensure the dataset itself is recorded
+        create_datasets_table(db_cursor, f"{table_prefix}{DATASETS_TABLE_SUFFIX}")
```

(N.B. you will also need to import `DATASETS_TABLE_SUFFIX` alongside the earlier import fix.)

This change keeps the registry in sync with the per‑dataset tables.

</blockquote></details>

</blockquote></details>

<details>
<summary>♻️ Duplicate comments (1)</summary><blockquote>

<details>
<summary>components/clp-py-utils/clp_py_utils/sql_table_schema_utils.py (1)</summary><blockquote>

`60-62`: _⚠️ Potential issue_

**Still using the wrong cursor variable – will raise `NameError`**  

The earlier review already pointed this out, but the fix has not been applied.  
`metadata_db_cursor` is undefined; use the function argument `db_cursor`.

```diff
-    metadata_db_cursor.execute(
+    db_cursor.execute(
```

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 01618f807702d269830a3a1b2be966b86eaee12b and 45258ce5d66a87304a82379d993a7024293e8e91.

</details>

<details>
<summary>📒 Files selected for processing (2)</summary>

* `components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py` (2 hunks)
* `components/clp-py-utils/clp_py_utils/sql_table_schema_utils.py` (1 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (1)</summary>

* components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (2)</summary>

* GitHub Check: lint-check (macos-latest)
* GitHub Check: lint-check (ubuntu-latest)

</details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T15:19:20Z
user: login: kirkrodrigues
body: For the PR title, how about:

```
refactor(clp-py-utils): Move CLP metadata DB table-creation logic into utility module; Add creation function for datasets table.
```
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-04-21T19:58:31Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>🔭 Outside diff range comments (1)</summary><blockquote>

<details>
<summary>components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py (1)</summary><blockquote>

`110-121`: _🛠️ Refactor suggestion_

**Consider adding datasets table creation to the main function.**

The `create_metadata_db_tables` function doesn't call `create_datasets_table` despite its name suggesting it creates all metadata tables. This seems inconsistent with the function's purpose of centralizing table creation.

Also, consider adding error handling for SQL execution and docstrings for this public function.

```diff
+"""
+Creates all necessary metadata tables for the CLP system.
+
+Args:
+    db_cursor: Database cursor for executing SQL statements
+    table_prefix: Prefix to apply to table names
+    dataset: Optional dataset name to create dataset-specific tables
+"""
 def create_metadata_db_tables(
     db_cursor: MariaDbCursor | MySQLCursor, table_prefix: str, dataset: str | None = None
 ) -> None:
+    try:
+        # Create datasets table regardless of dataset parameter
+        create_datasets_table(db_cursor, table_prefix)
+        
         if dataset is not None:
             table_prefix = f"{table_prefix}{dataset}_"
             _create_column_metadata_table(db_cursor, table_prefix)
         
         _create_archives_table(db_cursor, table_prefix)
         _create_tags_table(db_cursor, table_prefix)
         _create_archive_tags_table(db_cursor, table_prefix)
         _create_files_table(db_cursor, table_prefix)
+    except Exception as e:
+        # Consider appropriate error handling based on your application's needs
+        raise RuntimeError(f"Failed to create metadata tables: {str(e)}") from e
```

</blockquote></details>

</blockquote></details>

<details>
<summary>🧹 Nitpick comments (1)</summary><blockquote>

<details>
<summary>components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py (1)</summary><blockquote>

`98-108`: **Consider adding docstrings to public functions.**

While the implementation is good, adding docstrings would improve the maintainability and usability of this public function, explaining its purpose and parameters.

```diff
+"""
+Creates a table for storing dataset information.
+
+Args:
+    db_cursor: Database cursor for executing SQL statements
+    table_prefix: Prefix to apply to the table name
+"""
 def create_datasets_table(db_cursor: MariaDbCursor | MySQLCursor, table_prefix: str) -> None:
```

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 943020faae06d6aa26f7884a746d07e938db80b0 and 6d1b356ae94e792987dba38d73f5dba99c635d78.

</details>

<details>
<summary>📒 Files selected for processing (2)</summary>

* `components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py` (1 hunks)
* `components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py` (2 hunks)

</details>

<details>
<summary>✅ Files skipped from review due to trivial changes (1)</summary>

* components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (2)</summary>

* GitHub Check: lint-check (macos-latest)
* GitHub Check: lint-check (ubuntu-latest)

</details>

<details>
<summary>🔇 Additional comments (6)</summary><blockquote>

<details>
<summary>components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py (6)</summary>

`1-14`: **Appropriate imports and constants setup.**

The imports and constants are well-structured for the functionality provided. The file correctly uses type hints and imports cursor types from both MariaDB and MySQL for database compatibility.

---

`16-34`: **Well-structured table creation function for archives.**

The SQL query is properly formatted with appropriate columns, indexes, and constraints. Using f-strings for query construction with table prefixes is a clean approach.

---

`36-47`: **Clean implementation of tags table creation.**

The tags table schema is properly defined with appropriate constraints.

---

`49-61`: **Properly implemented join table with foreign key constraints.**

The archive_tags table correctly establishes relationships with appropriate references to the archives and tags tables.

---

`63-82`: **Files table creation looks good.**

The table schema includes appropriate columns and indexes. The `ROW_FORMAT=DYNAMIC` is a good choice given the large VARCHAR field for the path column.

---

`84-96`: **Column metadata table implementation is appropriate.**

The composite primary key on name and type is well-designed.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T22:49:28Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>🧹 Nitpick comments (2)</summary><blockquote>

<details>
<summary>components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py (2)</summary><blockquote>

`13-31`: **Archives table creation function includes all required fields and constraints.**

The function correctly defines the archives table structure with appropriate data types, constraints, and indexes. The use of parameterized table prefix enables flexibility for dataset-specific tables.


Consider adding a docstring to explain the function's purpose, parameters, and the table structure it creates. This would improve maintainability, especially for new developers.

```diff
 def _create_archives_table(db_cursor, table_prefix: str) -> None:
+    """
+    Creates the archives table if it doesn't exist.
+    
+    Args:
+        db_cursor: Database cursor for executing SQL
+        table_prefix: Prefix to prepend to the table name
+    
+    The archives table stores metadata about CLP archives including size, 
+    timestamps, and creation information.
+    """
     db_cursor.execute(
```

---

`60-79`: **Files table creation includes necessary fields and indexes.**

The function properly defines the files table structure with appropriate columns, indexes and primary key. The ROW_FORMAT=DYNAMIC setting is necessary to support the large VARCHAR field for the path.


The path field uses a very specific size (12288) and its index a different size (768). Consider extracting these as named constants to improve maintainability if these values are used elsewhere or have specific meaning in the context of the application.

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between ec106ba498efb083054155adec01ef7af589402e and ad67fd9e0113e7740ce006a4f4e088d88c865feb.

</details>

<details>
<summary>📒 Files selected for processing (4)</summary>

* `components/clp-package-utils/clp_package_utils/scripts/start_clp.py` (1 hunks)
* `components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py` (1 hunks)
* `components/clp-py-utils/clp_py_utils/create-db-tables.py` (2 hunks)
* `components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py` (2 hunks)

</details>

<details>
<summary>✅ Files skipped from review due to trivial changes (1)</summary>

* components/clp-package-utils/clp_package_utils/scripts/start_clp.py

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (1)</summary>

* components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🧬 Code Graph Analysis (1)</summary>

<details>
<summary>components/clp-py-utils/clp_py_utils/create-db-tables.py (1)</summary><blockquote>

<details>
<summary>components/clp-py-utils/clp_py_utils/clp_config.py (1)</summary>

* `StorageEngine` (55-57)

</details>

</blockquote></details>

</details>

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (3)</summary>

* GitHub Check: build (macos-latest)
* GitHub Check: lint-check (ubuntu-latest)
* GitHub Check: lint-check (macos-latest)

</details>

<details>
<summary>🔇 Additional comments (10)</summary><blockquote>

<details>
<summary>components/clp-py-utils/clp_py_utils/create-db-tables.py (4)</summary>

`7-7`: **Import statement added appropriately for the StorageEngine enum.**

The import of StorageEngine from clp_config aligns with the new command-line argument requirements and is correctly placed at the top of the file with other imports.

---

`23-29`: **Well-structured command-line argument for storage engine selection.**

The implementation properly:
- Makes the storage engine a required parameter
- Restricts choices to valid enum values
- Includes a helpful description for users

This change supports the refactoring of table creation logic into the utility module while ensuring proper validation of user input.

---

`33-33`: **Appropriate conversion of string argument to enum type.**

The code correctly converts the parsed string argument to a StorageEngine enum instance, ensuring type safety throughout the script's execution.

---

`41-41`: **Storage engine parameter properly passed to initialize script.**

The storage engine is correctly passed to the initialize-clp-metadata-db.py script, enabling dataset-specific table creation behaviour as described in the PR objectives.

Note that the storage engine is not passed to initialize-orchestration-db.py, which appears intentional as the orchestration database doesn't seem to require dataset-specific handling based on the PR objectives.

</details>
<details>
<summary>components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py (6)</summary>

`1-11`: **Clean import structure for a new utility module.**

The imports are appropriately organized with typing first, followed by project-specific imports. The use of imported constants from clp_config for table suffixes promotes consistency and maintainability across the codebase.

---

`33-44`: **Tags table creation function is well-structured.**

The function correctly implements the tags table creation with a primary key and a unique constraint on tag_name.

---

`46-58`: **Archive tags relationship table correctly implements foreign key constraints.**

The function properly creates a many-to-many relationship table between archives and tags with appropriate foreign key constraints referencing the parent tables. The constraints use the same table prefix, ensuring they work with dataset-specific tables.

---

`81-91`: **Column metadata table creation is correctly implemented.**

The function creates a simple column metadata table with a compound primary key for name and type. This aligns with the PR objectives, as this table is only created when a dataset is specified.

---

`93-103`: **Datasets table creation function is appropriately implemented as a public function.**

This function creates the datasets table to track dataset names and their archive storage directories. Making this a public function aligns with the PR objectives of supporting dataset-specific tables.

---

`105-114`: <details>
<summary>✅ Verification successful</summary>

**Well-designed coordinator function for creating metadata tables with dataset support.**

The `create_metadata_db_tables` function effectively:
1. Modifies the table prefix to include the dataset name when provided
2. Only creates the column metadata table when a dataset is specified (line 108)
3. Creates the core tables unconditionally

This implementation aligns perfectly with the PR objectives for dataset feature development.


Confirm if error handling should be added to handle potential database exceptions during table creation. Currently, any SQL errors would propagate to the caller. Run a quick check of where this function is called to see if errors are handled there:

---

🏁 Script executed:

```shell
#!/bin/bash
# Find calls to create_metadata_db_tables to check error handling
rg -A 3 "create_metadata_db_tables" --type py
```

Length of output: 1653

---



</details>

**Database exception handling is already implemented at the caller**

Calls to `create_metadata_db_tables` in `initialize-clp-metadata-db.py` are wrapped in a `try/except` that logs any SQL errors and rolls back accordingly, so you don’t need to add another layer of error handling inside `create_metadata_db_tables`.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T15:50:23Z
user: login: kirkrodrigues
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-04-23T02:24:18Z
user: login: kirkrodrigues
body: A few more suggestions.
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-04-23T17:27:17Z
user: login: Bill-hbrhbr
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-23T18:10:21Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>🔭 Outside diff range comments (1)</summary><blockquote>

<details>
<summary>components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py (1)</summary><blockquote>

`114-143`: _🛠️ Refactor suggestion_

**Consider adding error handling to the table creation process**

The function is well-documented and implements the functionality as described in the PR objectives. However, there's no error handling for database operations which could lead to incomplete table creation if an operation fails.


Consider adding try-except blocks to handle potential database errors:

```diff
def create_metadata_db_tables(db_cursor, table_prefix: str, dataset: str | None = None) -> None:
    """
    Creates the standard set of tables for CLP's metadata.

    :param db_cursor: The database cursor to execute the table creations.
    :param table_prefix: A string to prepend to all table names.
    :param dataset: If set, all tables will be named in a dataset-specific manner.
    """
    if dataset is not None:
        table_prefix = f"{table_prefix}{dataset}_"

    archives_table_name = f"{table_prefix}{ARCHIVES_TABLE_SUFFIX}"
    tags_table_name = f"{table_prefix}{TAGS_TABLE_SUFFIX}"
    archive_tags_table_name = f"{table_prefix}{ARCHIVE_TAGS_TABLE_SUFFIX}"

    # TODO: Update this to
    # {table_prefix}{CLP_DEFAULT_DATASET_NAME}_{COLUMN_METADATA_TABLE_SUFFIX} when we can also
    # change the indexer to match.
    column_metadata_table_name = (
        f"{table_prefix}{COLUMN_METADATA_TABLE_SUFFIX}_{CLP_DEFAULT_DATASET_NAME}"
    )

+    try:
        _create_archives_table(db_cursor, archives_table_name)
        _create_tags_table(db_cursor, tags_table_name)
        _create_archive_tags_table(
            db_cursor, archive_tags_table_name, archives_table_name, tags_table_name
        )
        _create_files_table(db_cursor, table_prefix)
        _create_column_metadata_table(db_cursor, column_metadata_table_name)
+    except Exception as e:
+        # Log the error or handle it appropriately
+        raise Exception(f"Error creating metadata tables: {str(e)}") from e
```

</blockquote></details>

</blockquote></details>

<details>
<summary>🧹 Nitpick comments (5)</summary><blockquote>

<details>
<summary>components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py (5)</summary><blockquote>

`14-32`: **Add docstring to helper function**

The `_create_archives_table` function lacks documentation. Consider adding a docstring to explain its purpose, parameters, and return value.


```diff
def _create_archives_table(db_cursor, archives_table_name: str) -> None:
+    """
+    Creates a table for storing archive metadata.
+
+    :param db_cursor: The database cursor to execute the table creation.
+    :param archives_table_name: The name of the table to create.
+    """
    db_cursor.execute(
        f"""
```

---

`34-45`: **Add docstring to helper function**

The `_create_tags_table` function lacks documentation. Consider adding a docstring to explain its purpose, parameters, and return value.


```diff
def _create_tags_table(db_cursor, tags_table_name: str) -> None:
+    """
+    Creates a table for storing tag information.
+
+    :param db_cursor: The database cursor to execute the table creation.
+    :param tags_table_name: The name of the table to create.
+    """
    db_cursor.execute(
        f"""
```

---

`47-61`: **Add docstring to helper function**

The `_create_archive_tags_table` function lacks documentation. Consider adding a docstring to explain its purpose, parameters, and return value.


```diff
def _create_archive_tags_table(
    db_cursor, archive_tags_table_name: str, archives_table_name: str, tags_table_name: str
) -> None:
+    """
+    Creates a table for storing relationships between archives and tags.
+
+    :param db_cursor: The database cursor to execute the table creation.
+    :param archive_tags_table_name: The name of the relation table to create.
+    :param archives_table_name: The name of the archives table to reference.
+    :param tags_table_name: The name of the tags table to reference.
+    """
    db_cursor.execute(
        f"""
```

---

`63-82`: **Add docstring and standardize parameters**

The `_create_files_table` function lacks documentation and uses `table_prefix` instead of a full table name like the other helper functions, which may cause confusion.


```diff
def _create_files_table(db_cursor, table_prefix: str) -> None:
+    """
+    Creates a table for storing file metadata.
+
+    :param db_cursor: The database cursor to execute the table creation.
+    :param table_prefix: A string to prepend to the files table suffix.
+    """
    db_cursor.execute(
        f"""
```

For consistency with other helper methods, consider refactoring to accept the full table name:

```diff
-def _create_files_table(db_cursor, table_prefix: str) -> None:
+def _create_files_table(db_cursor, files_table_name: str) -> None:
    """
    Creates a table for storing file metadata.

    :param db_cursor: The database cursor to execute the table creation.
-    :param table_prefix: A string to prepend to the files table suffix.
+    :param files_table_name: The name of the table to create.
    """
    db_cursor.execute(
        f"""
-        CREATE TABLE IF NOT EXISTS `{table_prefix}{FILES_TABLE_SUFFIX}` (
+        CREATE TABLE IF NOT EXISTS `{files_table_name}` (
```

Then update the call in `create_metadata_db_tables`:

```diff
-    _create_files_table(db_cursor, table_prefix)
+    files_table_name = f"{table_prefix}{FILES_TABLE_SUFFIX}"
+    _create_files_table(db_cursor, files_table_name)
```

---

`84-94`: **Add docstring to helper function**

The `_create_column_metadata_table` function lacks documentation. Consider adding a docstring to explain its purpose, parameters, and return value.


```diff
def _create_column_metadata_table(db_cursor, table_name: str) -> None:
+    """
+    Creates a table for storing column metadata.
+
+    :param db_cursor: The database cursor to execute the table creation.
+    :param table_name: The name of the table to create.
+    """
    db_cursor.execute(
        f"""
```

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between bc5ee6e84454be9526d77f78aad23670605b2ab2 and 9a82570750cd95920b8ab7342baec4e7670136f9.

</details>

<details>
<summary>📒 Files selected for processing (2)</summary>

* `components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py` (1 hunks)
* `components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py` (2 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (1)</summary>

* components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (2)</summary>

* GitHub Check: lint-check (ubuntu-latest)
* GitHub Check: lint-check (macos-latest)

</details>

<details>
<summary>🔇 Additional comments (3)</summary><blockquote>

<details>
<summary>components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py (3)</summary>

`1-11`: **Excellent use of future imports!**

The inclusion of `from __future__ import annotations` is perfect for supporting the Python 3.8 compatibility with the `|` syntax in type hints. This is aligned with a past review comment and ensures backward compatibility.

---

`96-112`: **Well-documented function implementation**

The `create_datasets_table` function is well-documented and properly implements the functionality described in the PR objectives for creating the datasets table.

---

`129-134`: **Ensure the TODO comment is tracked**

The TODO comment indicates future work needed on the column metadata table naming. Make sure this is tracked in your issue tracking system to avoid it being forgotten.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-23T18:13:28Z
user: login: kirkrodrigues
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-23T19:22:07Z

Review comments:
diff_hunk: @@ -203,6 +215,17 @@ def search_and_schedule_new_tasks(db_conn, db_cursor, clp_metadata_db_connection
             db_conn.commit()
             continue
 
+        dataset = input_config.dataset
+        if StorageEngine.CLP_S == clp_storage_engine and dataset not in seen_datasets:
+            seen_datasets.add(dataset)
+            query = f'INSERT INTO {table_prefix}{DATASETS_TABLE_SUFFIX} (name) VALUES ("{dataset}")'
+            db_cursor.execute(query)
+            create_archives_table(db_cursor, f"{table_prefix}{dataset}_{ARCHIVES_TABLE_SUFFIX}")
+            create_column_metadata_table(
+                db_cursor, f"{table_prefix}{dataset}_{COLUMN_METADATA_TABLE_SUFFIX}"
+            )
+            db_conn.commit()
+
path: components/job-orchestration/job_orchestration/scheduler/compress/compression_scheduler.py
user: login: coderabbitai[bot]
body: _⚠️ Potential issue_

**Fix SQL injection vulnerability and add error handling**

There are several issues with the dataset registration implementation:

1. SQL injection vulnerability: The dataset name is directly inserted into the query without parameter binding
2. Missing error handling for table creation failures
3. No validation of dataset names (length, special characters, etc.)


Implement parameter binding and add error handling:

```diff
 dataset = input_config.dataset
 if StorageEngine.CLP_S == clp_storage_engine and dataset not in seen_datasets:
     seen_datasets.add(dataset)
-    query = f'INSERT INTO {table_prefix}{DATASETS_TABLE_SUFFIX} (name) VALUES ("{dataset}")'
-    db_cursor.execute(query)
-    create_archives_table(db_cursor, f"{table_prefix}{dataset}_{ARCHIVES_TABLE_SUFFIX}")
-    create_column_metadata_table(
-        db_cursor, f"{table_prefix}{dataset}_{COLUMN_METADATA_TABLE_SUFFIX}"
-    )
-    db_conn.commit()
+    try:
+        # Validate dataset name (add appropriate validation logic)
+        if not re.match(r'^[a-zA-Z0-9_-]+$', dataset):
+            logger.error(f"Invalid dataset name: {dataset}. Using 'default' instead.")
+            dataset = "default"
+        
+        # Use parameter binding to prevent SQL injection
+        query = f"INSERT INTO {table_prefix}{DATASETS_TABLE_SUFFIX} (name) VALUES (%s)"
+        db_cursor.execute(query, (dataset,))
+        
+        # Create dataset-specific tables
+        create_archives_table(db_cursor, f"{table_prefix}{dataset}_{ARCHIVES_TABLE_SUFFIX}")
+        create_column_metadata_table(
+            db_cursor, f"{table_prefix}{dataset}_{COLUMN_METADATA_TABLE_SUFFIX}"
+        )
+        db_conn.commit()
+        logger.info(f"Created tables for new dataset: {dataset}")
+    except Exception as ex:
+        logger.error(f"Failed to create tables for dataset {dataset}: {ex}")
+        db_conn.rollback()
```

Don't forget to add the required import at the top of the file:
```python
import re
```

<!-- This is an auto-generated reply by CodeRabbit -->
created_at: 2025-04-17T09:55:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,6 +47,8 @@
 
 scheduled_jobs = {}
 
+seen_datasets = set()
path: components/job-orchestration/job_orchestration/scheduler/compress/compression_scheduler.py
user: login: coderabbitai[bot]
body: _🛠️ Refactor suggestion_

**Consider using database checks instead of in-memory tracking**

The `seen_datasets` global set will be lost if the process restarts, potentially causing redundant table creation attempts for datasets that were already processed. 


Consider querying the database to check if a dataset exists instead of relying on in-memory tracking:

```diff
-seen_datasets = set()
```

Then modify the dataset check logic to query the database instead:

```diff
-if StorageEngine.CLP_S == clp_storage_engine and dataset not in seen_datasets:
-    seen_datasets.add(dataset)
+if StorageEngine.CLP_S == clp_storage_engine:
+    # Check if dataset exists in the database
+    query = f"SELECT name FROM {table_prefix}{DATASETS_TABLE_SUFFIX} WHERE name = %s"
+    db_cursor.execute(query, (dataset,))
+    if not db_cursor.fetchone():
```


> Committable suggestion skipped: line range outside the PR's diff.

<!-- This is an auto-generated reply by CodeRabbit -->
created_at: 2025-04-17T09:55:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,6 +47,8 @@
 
 scheduled_jobs = {}
 
+seen_datasets = set()
path: components/job-orchestration/job_orchestration/scheduler/compress/compression_scheduler.py
user: login: Bill-hbrhbr
body: We've addressed the issue by updating in memory cache upon startup
created_at: 2025-04-17T16:54:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -203,6 +215,17 @@ def search_and_schedule_new_tasks(db_conn, db_cursor, clp_metadata_db_connection
             db_conn.commit()
             continue
 
+        dataset = input_config.dataset
+        if StorageEngine.CLP_S == clp_storage_engine and dataset not in seen_datasets:
+            seen_datasets.add(dataset)
+            query = f'INSERT INTO {table_prefix}{DATASETS_TABLE_SUFFIX} (name) VALUES ("{dataset}")'
+            db_cursor.execute(query)
+            create_archives_table(db_cursor, f"{table_prefix}{dataset}_{ARCHIVES_TABLE_SUFFIX}")
+            create_column_metadata_table(
+                db_cursor, f"{table_prefix}{dataset}_{COLUMN_METADATA_TABLE_SUFFIX}"
+            )
+            db_conn.commit()
+
path: components/job-orchestration/job_orchestration/scheduler/compress/compression_scheduler.py
user: login: Bill-hbrhbr
body: added a todo for checking `dataset` name is valid for sql tables
created_at: 2025-04-17T16:54:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,6 +47,8 @@
 
 scheduled_jobs = {}
 
+seen_datasets = set()
path: components/job-orchestration/job_orchestration/scheduler/compress/compression_scheduler.py
user: login: coderabbitai[bot]
body: @Bill-hbrhbr That's a great approach! Initializing the cache at startup gives you the performance benefits of in-memory checks while ensuring you don't lose track of datasets between restarts. Thanks for explaining how you addressed it.

🐰👍

---

<details>
<summary>✏️ Learnings added</summary>

```
Learnt from: Bill-hbrhbr
PR: y-scope/clp#831
File: components/job-orchestration/job_orchestration/scheduler/compress/compression_scheduler.py:0-0
Timestamp: 2025-04-17T16:55:06.632Z
Learning: In the compression scheduler, the team prefers initializing in-memory caches from the database at startup rather than performing repeated database queries for efficiency reasons. This approach maintains both performance and reliability across process restarts.
```

> Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure [review instructions](https://docs.coderabbit.ai/guides/review-instructions). You can manage existing learnings in the [app](https://app.coderabbit.ai/learnings).

</details>




<!-- <agentComponentType>chat-answer</agentComponentType> -->

<!-- <rawResChunk><answer>@Bill-hbrhbr That's a great approach! Initializing the cache at startup gives you the performance benefits of in-memory checks while ensuring you don't lose track of datasets between restarts. Thanks for explaining how you addressed it.

🐰👍

---

<details>
<summary>✏️ Learnings added</summary>

```
Learnt from: Bill-hbrhbr
PR: y-scope/clp#831
File: components/job-orchestration/job_orchestration/scheduler/compress/compression_scheduler.py:0-0
Timestamp: 2025-04-17T16:55:06.632Z
Learning: In the compression scheduler, the team prefers initializing in-memory caches from the database at startup rather than performing repeated database queries for efficiency reasons. This approach maintains both performance and reliability across process restarts.
```

> Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure [review instructions](https://docs.coderabbit.ai/guides/review-instructions). You can manage existing learnings in the [app](https://app.coderabbit.ai/learnings).

</details>

</answer></rawResChunk> -->

<!-- This is an auto-generated reply by CodeRabbit -->
created_at: 2025-04-17T16:55:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -203,6 +215,17 @@ def search_and_schedule_new_tasks(db_conn, db_cursor, clp_metadata_db_connection
             db_conn.commit()
             continue
 
+        dataset = input_config.dataset
+        if StorageEngine.CLP_S == clp_storage_engine and dataset not in seen_datasets:
+            seen_datasets.add(dataset)
+            query = f'INSERT INTO {table_prefix}{DATASETS_TABLE_SUFFIX} (name) VALUES ("{dataset}")'
+            db_cursor.execute(query)
+            create_archives_table(db_cursor, f"{table_prefix}{dataset}_{ARCHIVES_TABLE_SUFFIX}")
+            create_column_metadata_table(
+                db_cursor, f"{table_prefix}{dataset}_{COLUMN_METADATA_TABLE_SUFFIX}"
+            )
+            db_conn.commit()
+
path: components/job-orchestration/job_orchestration/scheduler/compress/compression_scheduler.py
user: login: coderabbitai[bot]
body: @Bill-hbrhbr Thank you for adding the TODO to check that dataset names are valid for SQL tables. That's a good step toward addressing the issue.

While the TODO will remind you to implement validation in the future, you might also want to consider addressing the SQL injection vulnerability sooner rather than later by using parameterized queries. This is a security best practice that prevents attackers from executing malicious SQL commands through user input.

A simple fix would be to change:
```python
query = f'INSERT INTO {table_prefix}{DATASETS_TABLE_SUFFIX} (name) VALUES ("{dataset}")'
db_cursor.execute(query)
```

To:
```python
query = f"INSERT INTO {table_prefix}{DATASETS_TABLE_SUFFIX} (name) VALUES (%s)"
db_cursor.execute(query, (dataset,))
```

Would you like me to show a more complete implementation that includes both validation and parameterized queries?

---

<details>
<summary>✏️ Learnings added</summary>

```
Learnt from: Bill-hbrhbr
PR: y-scope/clp#831
File: components/job-orchestration/job_orchestration/scheduler/compress/compression_scheduler.py:0-0
Timestamp: 2025-04-17T16:55:23.759Z
Learning: In the CLP project, SQL queries should use parameterized queries with placeholders (%s) and pass values as a tuple to `db_cursor.execute()` to prevent SQL injection, rather than directly interpolating values into the query string.
```

> Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure [review instructions](https://docs.coderabbit.ai/guides/review-instructions). You can manage existing learnings in the [app](https://app.coderabbit.ai/learnings).

</details>




<!-- <agentComponentType>chat-answer</agentComponentType> -->

<!-- <rawResChunk><answer>@Bill-hbrhbr Thank you for adding the TODO to check that dataset names are valid for SQL tables. That's a good step toward addressing the issue.

While the TODO will remind you to implement validation in the future, you might also want to consider addressing the SQL injection vulnerability sooner rather than later by using parameterized queries. This is a security best practice that prevents attackers from executing malicious SQL commands through user input.

A simple fix would be to change:
```python
query = f'INSERT INTO {table_prefix}{DATASETS_TABLE_SUFFIX} (name) VALUES ("{dataset}")'
db_cursor.execute(query)
```

To:
```python
query = f"INSERT INTO {table_prefix}{DATASETS_TABLE_SUFFIX} (name) VALUES (%s)"
db_cursor.execute(query, (dataset,))
```

Would you like me to show a more complete implementation that includes both validation and parameterized queries?

---

<details>
<summary>✏️ Learnings added</summary>

```
Learnt from: Bill-hbrhbr
PR: y-scope/clp#831
File: components/job-orchestration/job_orchestration/scheduler/compress/compression_scheduler.py:0-0
Timestamp: 2025-04-17T16:55:23.759Z
Learning: In the CLP project, SQL queries should use parameterized queries with placeholders (%s) and pass values as a tuple to `db_cursor.execute()` to prevent SQL injection, rather than directly interpolating values into the query string.
```

> Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure [review instructions](https://docs.coderabbit.ai/guides/review-instructions). You can manage existing learnings in the [app](https://app.coderabbit.ai/learnings).

</details>

</answer></rawResChunk> -->

<!-- This is an auto-generated reply by CodeRabbit -->
created_at: 2025-04-17T16:55:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -6,8 +6,16 @@
 
 from sql_adapter import SQL_Adapter
 
-from clp_py_utils.clp_config import Database
+from clp_py_utils.clp_config import (
+    ARCHIVE_TAGS_TABLE_SUFFIX,
+    ARCHIVES_TABLE_SUFFIX,
+    Database,
+    DATASETS_TABLE_SUFFIX,
+    FILES_TABLE_SUFFIX,
+    TAGS_TABLE_SUFFIX,
+)
 from clp_py_utils.core import read_yaml_config_file
+from clp_py_utils.sql_table_schema_utils import create_archives_table
 
path: components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py
user: login: coderabbitai[bot]
body: _⚠️ Potential issue_

**Import all helper functions you later invoke – otherwise you’ll hit `NameError` at runtime**  

Only `create_archives_table` is imported here, yet the code below calls `create_tags_table`, `create_archive_tags_table`, and `create_files_table`.  Because those symbols are not in this module’s namespace, the script will crash as soon as it reaches the first missing call.

```diff
-from clp_py_utils.sql_table_schema_utils import create_archives_table
+from clp_py_utils.sql_table_schema_utils import (
+    create_archives_table,
+    create_tags_table,
+    create_archive_tags_table,
+    create_files_table,
+    create_datasets_table,
+)
```

This also pre‑loads `create_datasets_table`, which you’ll need for the dataset work described in the PR objective.  



<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

`````suggestion
from clp_py_utils.clp_config import (
    ARCHIVE_TAGS_TABLE_SUFFIX,
    ARCHIVES_TABLE_SUFFIX,
    Database,
    DATASETS_TABLE_SUFFIX,
    FILES_TABLE_SUFFIX,
    TAGS_TABLE_SUFFIX,
)
from clp_py_utils.core import read_yaml_config_file
from clp_py_utils.sql_table_schema_utils import (
    create_archives_table,
    create_tags_table,
    create_archive_tags_table,
    create_files_table,
    create_datasets_table,
)
`````

</details>

<!-- suggestion_end -->

<!-- This is an auto-generated comment by CodeRabbit -->

✅ Addressed in commit fc6c126
created_at: 2025-04-21T13:49:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -35,76 +43,12 @@ def main(argv):
         with closing(sql_adapter.create_connection(True)) as metadata_db, closing(
             metadata_db.cursor(dictionary=True)
         ) as metadata_db_cursor:
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}archives` (
-                    `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
-                    `id` VARCHAR(64) NOT NULL,
-                    `begin_timestamp` BIGINT NOT NULL,
-                    `end_timestamp` BIGINT NOT NULL,
-                    `uncompressed_size` BIGINT NOT NULL,
-                    `size` BIGINT NOT NULL,
-                    `creator_id` VARCHAR(64) NOT NULL,
-                    `creation_ix` INT NOT NULL,
-                    KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
-                    UNIQUE KEY `archive_id` (`id`) USING BTREE,
-                    PRIMARY KEY (`pagination_id`)
-                )
-                """
+            create_archives_table(metadata_db_cursor, f"{table_prefix}{ARCHIVES_TABLE_SUFFIX}")
+            create_tags_table(metadata_db_cursor, f"{table_prefix}{TAGS_TABLE_SUFFIX}")
+            create_archive_tags_table(
+                metadata_db_cursor, f"{table_prefix}{ARCHIVE_TAGS_TABLE_SUFFIX}"
             )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}tags` (
-                    `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
-                    `tag_name` VARCHAR(255) NOT NULL,
-                    UNIQUE KEY (`tag_name`) USING BTREE,
-                    PRIMARY KEY (`tag_id`)
-                )
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}archive_tags` (
-                    `archive_id` VARCHAR(64) NOT NULL,
-                    `tag_id` INT unsigned NOT NULL,
-                    PRIMARY KEY (`archive_id`,`tag_id`),
-                    FOREIGN KEY (`archive_id`) REFERENCES `{table_prefix}archives` (`id`),
-                    FOREIGN KEY (`tag_id`) REFERENCES `{table_prefix}tags` (`tag_id`)
-                )
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}files` (
-                    `id` VARCHAR(64) NOT NULL,
-                    `orig_file_id` VARCHAR(64) NOT NULL,
-                    `path` VARCHAR(12288) NOT NULL,
-                    `begin_timestamp` BIGINT NOT NULL,
-                    `end_timestamp` BIGINT NOT NULL,
-                    `num_uncompressed_bytes` BIGINT NOT NULL,
-                    `begin_message_ix` BIGINT NOT NULL,
-                    `num_messages` BIGINT NOT NULL,
-                    `archive_id` VARCHAR(64) NOT NULL,
-                    KEY `files_path` (path(768)) USING BTREE,
-                    KEY `files_archive_id` (`archive_id`) USING BTREE,
-                    PRIMARY KEY (`id`)
-                ) ROW_FORMAT=DYNAMIC
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}column_metadata_default` (
-                    `name` VARCHAR(512) NOT NULL,
-                    `type` TINYINT NOT NULL,
-                    PRIMARY KEY (`name`, `type`)
-                )
-                """
-            )
-
+            create_files_table(metadata_db_cursor, f"{table_prefix}{FILES_TABLE_SUFFIX}")
             metadata_db.commit()
path: components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py
user: login: coderabbitai[bot]
body: _⚠️ Potential issue_

**Fix missing function definitions and create the datasets table**  

Because the helpers above are not (yet) imported, every call except `create_archives_table` will raise `NameError`.  In addition, the new `clp_datasets` table that the PR introduces is never created.

```diff
-            create_archives_table(metadata_db_cursor, f"{table_prefix}{ARCHIVES_TABLE_SUFFIX}")
-            create_tags_table(metadata_db_cursor, f"{table_prefix}{TAGS_TABLE_SUFFIX}")
-            create_archive_tags_table(
-                metadata_db_cursor, f"{table_prefix}{ARCHIVE_TAGS_TABLE_SUFFIX}"
-            )
-            create_files_table(metadata_db_cursor, f"{table_prefix}{FILES_TABLE_SUFFIX}")
+            create_archives_table(metadata_db_cursor, f"{table_prefix}{ARCHIVES_TABLE_SUFFIX}")
+            create_tags_table(metadata_db_cursor, f"{table_prefix}{TAGS_TABLE_SUFFIX}")
+            create_archive_tags_table(
+                metadata_db_cursor, f"{table_prefix}{ARCHIVE_TAGS_TABLE_SUFFIX}"
+            )
+            create_files_table(metadata_db_cursor, f"{table_prefix}{FILES_TABLE_SUFFIX}")
+            # New: ensure the datasets registry exists
+            create_datasets_table(metadata_db_cursor, f"{table_prefix}{DATASETS_TABLE_SUFFIX}")
```

This resolves the runtime failures and fulfils the PR’s stated objective of persisting dataset information.  



<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

`````suggestion
            create_archives_table(metadata_db_cursor, f"{table_prefix}{ARCHIVES_TABLE_SUFFIX}")
            create_tags_table(metadata_db_cursor, f"{table_prefix}{TAGS_TABLE_SUFFIX}")
            create_archive_tags_table(
                metadata_db_cursor, f"{table_prefix}{ARCHIVE_TAGS_TABLE_SUFFIX}"
            )
            create_files_table(metadata_db_cursor, f"{table_prefix}{FILES_TABLE_SUFFIX}")
            # New: ensure the datasets registry exists
            create_datasets_table(metadata_db_cursor, f"{table_prefix}{DATASETS_TABLE_SUFFIX}")
            metadata_db.commit()
`````

</details>

<!-- suggestion_end -->

<!-- This is an auto-generated comment by CodeRabbit -->

✅ Addressed in commit fc6c126
created_at: 2025-04-21T13:49:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,89 @@
+def create_archives_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
+            `id` VARCHAR(64) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `uncompressed_size` BIGINT NOT NULL,
+            `size` BIGINT NOT NULL,
+            `creator_id` VARCHAR(64) NOT NULL,
+            `creation_ix` INT NOT NULL,
+            KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
+            UNIQUE KEY `archive_id` (`id`) USING BTREE,
+            PRIMARY KEY (`pagination_id`)
+        )
+        """
+    )
+
+
+def create_tags_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
+            `tag_name` VARCHAR(255) NOT NULL,
+            UNIQUE KEY (`tag_name`) USING BTREE,
+            PRIMARY KEY (`tag_id`)
+        )
+        """
+    )
+
+
+def create_archive_tags_table(db_cursor, table_name: str) -> None:
+    metadata_db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `archive_id` VARCHAR(64) NOT NULL,
+            `tag_id` INT unsigned NOT NULL,
+            PRIMARY KEY (`archive_id`,`tag_id`),
+            FOREIGN KEY (`archive_id`) REFERENCES `{table_prefix}archives` (`id`),
+            FOREIGN KEY (`tag_id`) REFERENCES `{table_prefix}tags` (`tag_id`)
+        )
+        """
+    )
+
+
+def create_files_table(db_cursor, table_name: str) -> None:
+    metadata_db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `id` VARCHAR(64) NOT NULL,
+            `orig_file_id` VARCHAR(64) NOT NULL,
+            `path` VARCHAR(12288) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `num_uncompressed_bytes` BIGINT NOT NULL,
+            `begin_message_ix` BIGINT NOT NULL,
+            `num_messages` BIGINT NOT NULL,
+            `archive_id` VARCHAR(64) NOT NULL,
+            KEY `files_path` (path(768)) USING BTREE,
+            KEY `files_archive_id` (`archive_id`) USING BTREE,
+            PRIMARY KEY (`id`)
+        ) ROW_FORMAT=DYNAMIC
+        """
+    )
path: components/clp-py-utils/clp_py_utils/sql_table_schema_utils.py
user: login: coderabbitai[bot]
body: _⚠️ Potential issue_

**Same cursor bug repeats here – update to `db_cursor`**  

`create_files_table` also references `metadata_db_cursor`; switch to the function argument to avoid a `NameError`.

```diff
-def create_files_table(db_cursor, table_name: str) -> None:
-    metadata_db_cursor.execute(
+def create_files_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
```

No other changes required.  



<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

`````suggestion
def create_files_table(db_cursor, table_name: str) -> None:
    db_cursor.execute(
        f"""
        CREATE TABLE IF NOT EXISTS `{table_name}` (
            `id` VARCHAR(64) NOT NULL,
            `orig_file_id` VARCHAR(64) NOT NULL,
            `path` VARCHAR(12288) NOT NULL,
            `begin_timestamp` BIGINT NOT NULL,
            `end_timestamp` BIGINT NOT NULL,
            `num_uncompressed_bytes` BIGINT NOT NULL,
            `begin_message_ix` BIGINT NOT NULL,
            `num_messages` BIGINT NOT NULL,
            `archive_id` VARCHAR(64) NOT NULL,
            KEY `files_path` (path(768)) USING BTREE,
            KEY `files_archive_id` (`archive_id`) USING BTREE,
            PRIMARY KEY (`id`)
        ) ROW_FORMAT=DYNAMIC
        """
    )
`````

</details>

<!-- suggestion_end -->

<!-- This is an auto-generated comment by CodeRabbit -->

✅ Addressed in commit c8bf33d
created_at: 2025-04-21T13:49:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,89 @@
+def create_archives_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
+            `id` VARCHAR(64) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `uncompressed_size` BIGINT NOT NULL,
+            `size` BIGINT NOT NULL,
+            `creator_id` VARCHAR(64) NOT NULL,
+            `creation_ix` INT NOT NULL,
+            KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
+            UNIQUE KEY `archive_id` (`id`) USING BTREE,
+            PRIMARY KEY (`pagination_id`)
+        )
+        """
+    )
+
+
+def create_tags_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
+            `tag_name` VARCHAR(255) NOT NULL,
+            UNIQUE KEY (`tag_name`) USING BTREE,
+            PRIMARY KEY (`tag_id`)
+        )
+        """
+    )
+
+
+def create_archive_tags_table(db_cursor, table_name: str) -> None:
+    metadata_db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `archive_id` VARCHAR(64) NOT NULL,
+            `tag_id` INT unsigned NOT NULL,
+            PRIMARY KEY (`archive_id`,`tag_id`),
+            FOREIGN KEY (`archive_id`) REFERENCES `{table_prefix}archives` (`id`),
+            FOREIGN KEY (`tag_id`) REFERENCES `{table_prefix}tags` (`tag_id`)
+        )
+        """
+    )
path: components/clp-py-utils/clp_py_utils/sql_table_schema_utils.py
user: login: coderabbitai[bot]
body: _⚠️ Potential issue_

**Use the correct cursor parameter and derive FK targets without relying on an undefined `table_prefix`**  

`metadata_db_cursor` and `table_prefix` are not defined inside this function. Their use will raise `NameError`.  Replace them with the `db_cursor` argument and compute the referenced table names from `table_name`, mirroring the earlier implementation you removed.

```diff
-def create_archive_tags_table(db_cursor, table_name: str) -> None:
-    metadata_db_cursor.execute(
+def create_archive_tags_table(db_cursor, table_name: str) -> None:
     db_cursor.execute(
         f"""
         CREATE TABLE IF NOT EXISTS `{table_name}` (
             `archive_id` VARCHAR(64) NOT NULL,
             `tag_id` INT unsigned NOT NULL,
             PRIMARY KEY (`archive_id`,`tag_id`),
-            FOREIGN KEY (`archive_id`) REFERENCES `{table_prefix}archives` (`id`),
-            FOREIGN KEY (`tag_id`) REFERENCES `{table_prefix}tags` (`tag_id`)
+            FOREIGN KEY (`archive_id`) REFERENCES `{table_name.replace('_archive_tags','_archives')}` (`id`),
+            FOREIGN KEY (`tag_id`) REFERENCES `{table_name.replace('_archive_tags','_tags')}` (`tag_id`)
         )
         """
     )
```

Without this fix, table creation will fail immediately.  



<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

`````suggestion
def create_archive_tags_table(db_cursor, table_name: str) -> None:
    db_cursor.execute(
        f"""
        CREATE TABLE IF NOT EXISTS `{table_name}` (
            `archive_id` VARCHAR(64) NOT NULL,
            `tag_id` INT unsigned NOT NULL,
            PRIMARY KEY (`archive_id`,`tag_id`),
            FOREIGN KEY (`archive_id`) REFERENCES `{table_name.replace('_archive_tags','_archives')}` (`id`),
            FOREIGN KEY (`tag_id`) REFERENCES `{table_name.replace('_archive_tags','_tags')}` (`tag_id`)
        )
        """
    )
`````

</details>

<!-- suggestion_end -->

<!-- This is an auto-generated comment by CodeRabbit -->

✅ Addressed in commit fc6c126
created_at: 2025-04-21T13:49:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,122 @@
+import typing
+
+from clp_py_utils.clp_config import (
+    ARCHIVE_TAGS_TABLE_SUFFIX,
+    ARCHIVES_TABLE_SUFFIX,
+    FILES_TABLE_SUFFIX,
+    TAGS_TABLE_SUFFIX,
+)
path: components/clp-py-utils/clp_py_utils/sql_table_schema_utils.py
user: login: coderabbitai[bot]
body: _⚠️ Potential issue_

**Missing constant import will raise `NameError` at runtime**

`create_metadata_db_tables` (see l.110) references `COLUMN_METADATA_TABLE_SUFFIX`, but this constant is not imported here.  When that branch executes, the function will crash with `NameError: name 'COLUMN_METADATA_TABLE_SUFFIX' is not defined`.

```diff
 from clp_py_utils.clp_config import (
     ARCHIVE_TAGS_TABLE_SUFFIX,
     ARCHIVES_TABLE_SUFFIX,
     FILES_TABLE_SUFFIX,
     TAGS_TABLE_SUFFIX,
+    COLUMN_METADATA_TABLE_SUFFIX,
 )
```



<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

`````suggestion
from clp_py_utils.clp_config import (
    ARCHIVE_TAGS_TABLE_SUFFIX,
    ARCHIVES_TABLE_SUFFIX,
    FILES_TABLE_SUFFIX,
    TAGS_TABLE_SUFFIX,
    COLUMN_METADATA_TABLE_SUFFIX,
)
`````

</details>

<!-- suggestion_end -->

<!-- This is an auto-generated comment by CodeRabbit -->

✅ Addressed in commits c8bf33d to 943020f
created_at: 2025-04-21T15:19:19Z
author_association: CONTRIBUTOR
path: components/clp-py-utils/clp_py_utils/sql_table_schema_utils.py
user: login: kirkrodrigues
body: How about `clp_metadata_db_utils.py`?
created_at: 2025-04-21T19:46:55Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,123 @@
+import typing
+
+from clp_py_utils.clp_config import (
+    ARCHIVE_TAGS_TABLE_SUFFIX,
+    ARCHIVES_TABLE_SUFFIX,
+    COLUMN_METADATA_TABLE_SUFFIX,
+    FILES_TABLE_SUFFIX,
+    TAGS_TABLE_SUFFIX,
+)
+
+
+def create_archives_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
+            `id` VARCHAR(64) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `uncompressed_size` BIGINT NOT NULL,
+            `size` BIGINT NOT NULL,
+            `creator_id` VARCHAR(64) NOT NULL,
+            `creation_ix` INT NOT NULL,
+            KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
+            UNIQUE KEY `archive_id` (`id`) USING BTREE,
+            PRIMARY KEY (`pagination_id`)
+        )
+        """
+    )
+
+
+def create_tags_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
+            `tag_name` VARCHAR(255) NOT NULL,
+            UNIQUE KEY (`tag_name`) USING BTREE,
+            PRIMARY KEY (`tag_id`)
+        )
+        """
+    )
+
+
+def create_archive_tags_table(
+    db_cursor, table_name: str, archives_table_name: str, tags_table_name: str
+) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `archive_id` VARCHAR(64) NOT NULL,
+            `tag_id` INT unsigned NOT NULL,
+            PRIMARY KEY (`archive_id`,`tag_id`),
+            FOREIGN KEY (`archive_id`) REFERENCES `{archives_table_name}` (`id`),
+            FOREIGN KEY (`tag_id`) REFERENCES `{tags_table_name}` (`tag_id`)
+        )
+        """
+    )
+
+
+def create_files_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `id` VARCHAR(64) NOT NULL,
+            `orig_file_id` VARCHAR(64) NOT NULL,
+            `path` VARCHAR(12288) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `num_uncompressed_bytes` BIGINT NOT NULL,
+            `begin_message_ix` BIGINT NOT NULL,
+            `num_messages` BIGINT NOT NULL,
+            `archive_id` VARCHAR(64) NOT NULL,
+            KEY `files_path` (path(768)) USING BTREE,
+            KEY `files_archive_id` (`archive_id`) USING BTREE,
+            PRIMARY KEY (`id`)
+        ) ROW_FORMAT=DYNAMIC
+        """
+    )
+
+
+def create_datasets_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `name` VARCHAR(255) NOT NULL,
+            `archive_storage_directory` VARCHAR(4096) NOT NULL,
+            PRIMARY KEY (`name`)
+        )
+        """
+    )
+
+
+def create_column_metadata_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `name` VARCHAR(512) NOT NULL,
+            `type` TINYINT NOT NULL,
+            PRIMARY KEY (`name`, `type`)
+        )
+        """
+    )
+
+
+def create_metadata_db_tables(
+    db_cursor, table_prefix: str, dataset: typing.Optional[str] = None
+) -> None:
+    if dataset is not None:
+        table_prefix = f"{table_prefix}{dataset}_"
+        create_column_metadata_table(db_cursor, f"{table_prefix}{COLUMN_METADATA_TABLE_SUFFIX}")
+
+    archives_table_name = f"{table_prefix}{ARCHIVES_TABLE_SUFFIX}"
+    tags_table_name = f"{table_prefix}{TAGS_TABLE_SUFFIX}"
+    archive_tags_table_name = f"{table_prefix}{ARCHIVE_TAGS_TABLE_SUFFIX}"
+    files_table_name = f"{table_prefix}{FILES_TABLE_SUFFIX}"
+
+    create_archives_table(db_cursor, archives_table_name)
+    create_tags_table(db_cursor, tags_table_name)
+    create_archive_tags_table(
+        db_cursor, archive_tags_table_name, archives_table_name, tags_table_name
+    )
path: components/clp-py-utils/clp_py_utils/sql_table_schema_utils.py
user: login: kirkrodrigues
body: Since the order of these function calls matters, how about making each of them private (prefix with `_`) and then creating a single function that calls all three in the current order?
created_at: 2025-04-21T19:47:51Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,123 @@
+import typing
+
+from clp_py_utils.clp_config import (
+    ARCHIVE_TAGS_TABLE_SUFFIX,
+    ARCHIVES_TABLE_SUFFIX,
+    COLUMN_METADATA_TABLE_SUFFIX,
+    FILES_TABLE_SUFFIX,
+    TAGS_TABLE_SUFFIX,
+)
+
+
+def create_archives_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
+            `id` VARCHAR(64) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `uncompressed_size` BIGINT NOT NULL,
+            `size` BIGINT NOT NULL,
+            `creator_id` VARCHAR(64) NOT NULL,
+            `creation_ix` INT NOT NULL,
+            KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
+            UNIQUE KEY `archive_id` (`id`) USING BTREE,
+            PRIMARY KEY (`pagination_id`)
+        )
+        """
+    )
+
+
+def create_tags_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
+            `tag_name` VARCHAR(255) NOT NULL,
+            UNIQUE KEY (`tag_name`) USING BTREE,
+            PRIMARY KEY (`tag_id`)
+        )
+        """
+    )
+
+
+def create_archive_tags_table(
+    db_cursor, table_name: str, archives_table_name: str, tags_table_name: str
+) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `archive_id` VARCHAR(64) NOT NULL,
+            `tag_id` INT unsigned NOT NULL,
+            PRIMARY KEY (`archive_id`,`tag_id`),
+            FOREIGN KEY (`archive_id`) REFERENCES `{archives_table_name}` (`id`),
+            FOREIGN KEY (`tag_id`) REFERENCES `{tags_table_name}` (`tag_id`)
+        )
+        """
+    )
+
+
+def create_files_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `id` VARCHAR(64) NOT NULL,
+            `orig_file_id` VARCHAR(64) NOT NULL,
+            `path` VARCHAR(12288) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `num_uncompressed_bytes` BIGINT NOT NULL,
+            `begin_message_ix` BIGINT NOT NULL,
+            `num_messages` BIGINT NOT NULL,
+            `archive_id` VARCHAR(64) NOT NULL,
+            KEY `files_path` (path(768)) USING BTREE,
+            KEY `files_archive_id` (`archive_id`) USING BTREE,
+            PRIMARY KEY (`id`)
+        ) ROW_FORMAT=DYNAMIC
+        """
+    )
+
+
+def create_datasets_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `name` VARCHAR(255) NOT NULL,
+            `archive_storage_directory` VARCHAR(4096) NOT NULL,
+            PRIMARY KEY (`name`)
+        )
+        """
+    )
+
+
+def create_column_metadata_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `name` VARCHAR(512) NOT NULL,
+            `type` TINYINT NOT NULL,
+            PRIMARY KEY (`name`, `type`)
+        )
+        """
+    )
+
+
+def create_metadata_db_tables(
+    db_cursor, table_prefix: str, dataset: typing.Optional[str] = None
path: components/clp-py-utils/clp_py_utils/sql_table_schema_utils.py
user: login: kirkrodrigues
body: Can we add the type for the db_cursor? If I'm not mistaken, it's `Cursor | MySQLCursor` where `Cursor` can be imported from `mariadb` and `MySQLCursor` can be imported from `mysql.connector.cursor`.
created_at: 2025-04-21T19:48:41Z
author_association: MEMBER
path: components/clp-py-utils/clp_py_utils/sql_table_schema_utils.py
user: login: kirkrodrigues
body: Since this is a new file, let's follow best practices:

* Add doc strings for all public functions and any non-trivial private functions.
* Prefix any private function name with `_`.
created_at: 2025-04-21T19:50:07Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,123 @@
+import typing
+
+from clp_py_utils.clp_config import (
+    ARCHIVE_TAGS_TABLE_SUFFIX,
+    ARCHIVES_TABLE_SUFFIX,
+    COLUMN_METADATA_TABLE_SUFFIX,
+    FILES_TABLE_SUFFIX,
+    TAGS_TABLE_SUFFIX,
+)
+
+
+def create_archives_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
+            `id` VARCHAR(64) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `uncompressed_size` BIGINT NOT NULL,
+            `size` BIGINT NOT NULL,
+            `creator_id` VARCHAR(64) NOT NULL,
+            `creation_ix` INT NOT NULL,
+            KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
+            UNIQUE KEY `archive_id` (`id`) USING BTREE,
+            PRIMARY KEY (`pagination_id`)
+        )
+        """
+    )
+
+
+def create_tags_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
+            `tag_name` VARCHAR(255) NOT NULL,
+            UNIQUE KEY (`tag_name`) USING BTREE,
+            PRIMARY KEY (`tag_id`)
+        )
+        """
+    )
+
+
+def create_archive_tags_table(
+    db_cursor, table_name: str, archives_table_name: str, tags_table_name: str
+) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `archive_id` VARCHAR(64) NOT NULL,
+            `tag_id` INT unsigned NOT NULL,
+            PRIMARY KEY (`archive_id`,`tag_id`),
+            FOREIGN KEY (`archive_id`) REFERENCES `{archives_table_name}` (`id`),
+            FOREIGN KEY (`tag_id`) REFERENCES `{tags_table_name}` (`tag_id`)
+        )
+        """
+    )
+
+
+def create_files_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `id` VARCHAR(64) NOT NULL,
+            `orig_file_id` VARCHAR(64) NOT NULL,
+            `path` VARCHAR(12288) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `num_uncompressed_bytes` BIGINT NOT NULL,
+            `begin_message_ix` BIGINT NOT NULL,
+            `num_messages` BIGINT NOT NULL,
+            `archive_id` VARCHAR(64) NOT NULL,
+            KEY `files_path` (path(768)) USING BTREE,
+            KEY `files_archive_id` (`archive_id`) USING BTREE,
+            PRIMARY KEY (`id`)
+        ) ROW_FORMAT=DYNAMIC
+        """
+    )
+
+
+def create_datasets_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `name` VARCHAR(255) NOT NULL,
+            `archive_storage_directory` VARCHAR(4096) NOT NULL,
+            PRIMARY KEY (`name`)
+        )
+        """
+    )
+
+
+def create_column_metadata_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `name` VARCHAR(512) NOT NULL,
+            `type` TINYINT NOT NULL,
+            PRIMARY KEY (`name`, `type`)
+        )
+        """
+    )
+
+
+def create_metadata_db_tables(
+    db_cursor, table_prefix: str, dataset: typing.Optional[str] = None
+) -> None:
+    if dataset is not None:
+        table_prefix = f"{table_prefix}{dataset}_"
+        create_column_metadata_table(db_cursor, f"{table_prefix}{COLUMN_METADATA_TABLE_SUFFIX}")
+
+    archives_table_name = f"{table_prefix}{ARCHIVES_TABLE_SUFFIX}"
+    tags_table_name = f"{table_prefix}{TAGS_TABLE_SUFFIX}"
+    archive_tags_table_name = f"{table_prefix}{ARCHIVE_TAGS_TABLE_SUFFIX}"
+    files_table_name = f"{table_prefix}{FILES_TABLE_SUFFIX}"
+
+    create_archives_table(db_cursor, archives_table_name)
+    create_tags_table(db_cursor, tags_table_name)
+    create_archive_tags_table(
+        db_cursor, archive_tags_table_name, archives_table_name, tags_table_name
+    )
path: components/clp-py-utils/clp_py_utils/sql_table_schema_utils.py
user: login: kirkrodrigues
body: Unless we anticipate creating tables with varying suffixes in the future, I think it might make more sense to pass in the table_prefix in to each function, and have the function add the suffix relevant itself.
created_at: 2025-04-21T19:51:32Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,123 @@
+import typing
+
+from clp_py_utils.clp_config import (
+    ARCHIVE_TAGS_TABLE_SUFFIX,
+    ARCHIVES_TABLE_SUFFIX,
+    COLUMN_METADATA_TABLE_SUFFIX,
+    FILES_TABLE_SUFFIX,
+    TAGS_TABLE_SUFFIX,
+)
+
+
+def create_archives_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
+            `id` VARCHAR(64) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `uncompressed_size` BIGINT NOT NULL,
+            `size` BIGINT NOT NULL,
+            `creator_id` VARCHAR(64) NOT NULL,
+            `creation_ix` INT NOT NULL,
+            KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
+            UNIQUE KEY `archive_id` (`id`) USING BTREE,
+            PRIMARY KEY (`pagination_id`)
+        )
+        """
+    )
+
+
+def create_tags_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
+            `tag_name` VARCHAR(255) NOT NULL,
+            UNIQUE KEY (`tag_name`) USING BTREE,
+            PRIMARY KEY (`tag_id`)
+        )
+        """
+    )
+
+
+def create_archive_tags_table(
+    db_cursor, table_name: str, archives_table_name: str, tags_table_name: str
+) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `archive_id` VARCHAR(64) NOT NULL,
+            `tag_id` INT unsigned NOT NULL,
+            PRIMARY KEY (`archive_id`,`tag_id`),
+            FOREIGN KEY (`archive_id`) REFERENCES `{archives_table_name}` (`id`),
+            FOREIGN KEY (`tag_id`) REFERENCES `{tags_table_name}` (`tag_id`)
+        )
+        """
+    )
+
+
+def create_files_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `id` VARCHAR(64) NOT NULL,
+            `orig_file_id` VARCHAR(64) NOT NULL,
+            `path` VARCHAR(12288) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `num_uncompressed_bytes` BIGINT NOT NULL,
+            `begin_message_ix` BIGINT NOT NULL,
+            `num_messages` BIGINT NOT NULL,
+            `archive_id` VARCHAR(64) NOT NULL,
+            KEY `files_path` (path(768)) USING BTREE,
+            KEY `files_archive_id` (`archive_id`) USING BTREE,
+            PRIMARY KEY (`id`)
+        ) ROW_FORMAT=DYNAMIC
+        """
+    )
+
+
+def create_datasets_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `name` VARCHAR(255) NOT NULL,
+            `archive_storage_directory` VARCHAR(4096) NOT NULL,
+            PRIMARY KEY (`name`)
+        )
+        """
+    )
+
+
+def create_column_metadata_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `name` VARCHAR(512) NOT NULL,
+            `type` TINYINT NOT NULL,
+            PRIMARY KEY (`name`, `type`)
+        )
+        """
+    )
+
+
+def create_metadata_db_tables(
+    db_cursor, table_prefix: str, dataset: typing.Optional[str] = None
+) -> None:
+    if dataset is not None:
+        table_prefix = f"{table_prefix}{dataset}_"
+        create_column_metadata_table(db_cursor, f"{table_prefix}{COLUMN_METADATA_TABLE_SUFFIX}")
path: components/clp-py-utils/clp_py_utils/sql_table_schema_utils.py
user: login: kirkrodrigues
body: To make the creation of the column_metadata and datasets table clp-s specific, how about passing a storage-engine argument to `create-db-tables.py`, which in turn will pass it to `initialize-clp-metadata-db.py`, which in turn will pass it into these functions?
created_at: 2025-04-21T19:55:03Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,125 @@
+import typing
path: components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py
user: login: kirkrodrigues
body: ```suggestion
```
created_at: 2025-04-23T02:02:56Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,125 @@
+import typing
+
+from clp_py_utils.clp_config import (
+    ARCHIVE_TAGS_TABLE_SUFFIX,
+    ARCHIVES_TABLE_SUFFIX,
+    COLUMN_METADATA_TABLE_SUFFIX,
+    DATASETS_TABLE_SUFFIX,
+    FILES_TABLE_SUFFIX,
+    TAGS_TABLE_SUFFIX,
+)
+
+
+def _create_archives_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{ARCHIVES_TABLE_SUFFIX}` (
+            `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
+            `id` VARCHAR(64) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `uncompressed_size` BIGINT NOT NULL,
+            `size` BIGINT NOT NULL,
+            `creator_id` VARCHAR(64) NOT NULL,
+            `creation_ix` INT NOT NULL,
+            KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
+            UNIQUE KEY `archive_id` (`id`) USING BTREE,
+            PRIMARY KEY (`pagination_id`)
+        )
+        """
+    )
+
+
+def _create_tags_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{TAGS_TABLE_SUFFIX}` (
+            `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
+            `tag_name` VARCHAR(255) NOT NULL,
+            UNIQUE KEY (`tag_name`) USING BTREE,
+            PRIMARY KEY (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_archive_tags_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{ARCHIVE_TAGS_TABLE_SUFFIX}` (
+            `archive_id` VARCHAR(64) NOT NULL,
+            `tag_id` INT unsigned NOT NULL,
+            PRIMARY KEY (`archive_id`,`tag_id`),
+            FOREIGN KEY (`archive_id`) REFERENCES `{table_prefix}{ARCHIVES_TABLE_SUFFIX}` (`id`),
+            FOREIGN KEY (`tag_id`) REFERENCES `{table_prefix}{TAGS_TABLE_SUFFIX}` (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_files_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{FILES_TABLE_SUFFIX}` (
+            `id` VARCHAR(64) NOT NULL,
+            `orig_file_id` VARCHAR(64) NOT NULL,
+            `path` VARCHAR(12288) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `num_uncompressed_bytes` BIGINT NOT NULL,
+            `begin_message_ix` BIGINT NOT NULL,
+            `num_messages` BIGINT NOT NULL,
+            `archive_id` VARCHAR(64) NOT NULL,
+            KEY `files_path` (path(768)) USING BTREE,
+            KEY `files_archive_id` (`archive_id`) USING BTREE,
+            PRIMARY KEY (`id`)
+        ) ROW_FORMAT=DYNAMIC
+        """
+    )
+
+
+def _create_column_metadata_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{COLUMN_METADATA_TABLE_SUFFIX}` (
+            `name` VARCHAR(512) NOT NULL,
+            `type` TINYINT NOT NULL,
+            PRIMARY KEY (`name`, `type`)
+        )
+        """
+    )
+
+
+def create_datasets_table(db_cursor, table_prefix: str) -> None:
+    """
+    Creates the table to store the datasets information (for clp-s only).
path: components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py
user: login: kirkrodrigues
body: ```suggestion
    Creates the dataset information table.

```
created_at: 2025-04-23T02:03:43Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,125 @@
+import typing
+
+from clp_py_utils.clp_config import (
+    ARCHIVE_TAGS_TABLE_SUFFIX,
+    ARCHIVES_TABLE_SUFFIX,
+    COLUMN_METADATA_TABLE_SUFFIX,
+    DATASETS_TABLE_SUFFIX,
+    FILES_TABLE_SUFFIX,
+    TAGS_TABLE_SUFFIX,
+)
+
+
+def _create_archives_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{ARCHIVES_TABLE_SUFFIX}` (
+            `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
+            `id` VARCHAR(64) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `uncompressed_size` BIGINT NOT NULL,
+            `size` BIGINT NOT NULL,
+            `creator_id` VARCHAR(64) NOT NULL,
+            `creation_ix` INT NOT NULL,
+            KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
+            UNIQUE KEY `archive_id` (`id`) USING BTREE,
+            PRIMARY KEY (`pagination_id`)
+        )
+        """
+    )
+
+
+def _create_tags_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{TAGS_TABLE_SUFFIX}` (
+            `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
+            `tag_name` VARCHAR(255) NOT NULL,
+            UNIQUE KEY (`tag_name`) USING BTREE,
+            PRIMARY KEY (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_archive_tags_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{ARCHIVE_TAGS_TABLE_SUFFIX}` (
+            `archive_id` VARCHAR(64) NOT NULL,
+            `tag_id` INT unsigned NOT NULL,
+            PRIMARY KEY (`archive_id`,`tag_id`),
+            FOREIGN KEY (`archive_id`) REFERENCES `{table_prefix}{ARCHIVES_TABLE_SUFFIX}` (`id`),
+            FOREIGN KEY (`tag_id`) REFERENCES `{table_prefix}{TAGS_TABLE_SUFFIX}` (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_files_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{FILES_TABLE_SUFFIX}` (
+            `id` VARCHAR(64) NOT NULL,
+            `orig_file_id` VARCHAR(64) NOT NULL,
+            `path` VARCHAR(12288) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `num_uncompressed_bytes` BIGINT NOT NULL,
+            `begin_message_ix` BIGINT NOT NULL,
+            `num_messages` BIGINT NOT NULL,
+            `archive_id` VARCHAR(64) NOT NULL,
+            KEY `files_path` (path(768)) USING BTREE,
+            KEY `files_archive_id` (`archive_id`) USING BTREE,
+            PRIMARY KEY (`id`)
+        ) ROW_FORMAT=DYNAMIC
+        """
+    )
+
+
+def _create_column_metadata_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{COLUMN_METADATA_TABLE_SUFFIX}` (
+            `name` VARCHAR(512) NOT NULL,
+            `type` TINYINT NOT NULL,
+            PRIMARY KEY (`name`, `type`)
+        )
+        """
+    )
+
+
+def create_datasets_table(db_cursor, table_prefix: str) -> None:
+    """
+    Creates the table to store the datasets information (for clp-s only).
+    :param db_cursor: The database cursor to execute the table creation.
+    :param table_prefix: A string prefix applied to the dataset table name.
path: components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py
user: login: kirkrodrigues
body: ```suggestion
    :param table_prefix: A string to prepend to the table name.
```
created_at: 2025-04-23T02:03:51Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,125 @@
+import typing
+
+from clp_py_utils.clp_config import (
+    ARCHIVE_TAGS_TABLE_SUFFIX,
+    ARCHIVES_TABLE_SUFFIX,
+    COLUMN_METADATA_TABLE_SUFFIX,
+    DATASETS_TABLE_SUFFIX,
+    FILES_TABLE_SUFFIX,
+    TAGS_TABLE_SUFFIX,
+)
+
+
+def _create_archives_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{ARCHIVES_TABLE_SUFFIX}` (
+            `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
+            `id` VARCHAR(64) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `uncompressed_size` BIGINT NOT NULL,
+            `size` BIGINT NOT NULL,
+            `creator_id` VARCHAR(64) NOT NULL,
+            `creation_ix` INT NOT NULL,
+            KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
+            UNIQUE KEY `archive_id` (`id`) USING BTREE,
+            PRIMARY KEY (`pagination_id`)
+        )
+        """
+    )
+
+
+def _create_tags_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{TAGS_TABLE_SUFFIX}` (
+            `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
+            `tag_name` VARCHAR(255) NOT NULL,
+            UNIQUE KEY (`tag_name`) USING BTREE,
+            PRIMARY KEY (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_archive_tags_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{ARCHIVE_TAGS_TABLE_SUFFIX}` (
+            `archive_id` VARCHAR(64) NOT NULL,
+            `tag_id` INT unsigned NOT NULL,
+            PRIMARY KEY (`archive_id`,`tag_id`),
+            FOREIGN KEY (`archive_id`) REFERENCES `{table_prefix}{ARCHIVES_TABLE_SUFFIX}` (`id`),
+            FOREIGN KEY (`tag_id`) REFERENCES `{table_prefix}{TAGS_TABLE_SUFFIX}` (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_files_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{FILES_TABLE_SUFFIX}` (
+            `id` VARCHAR(64) NOT NULL,
+            `orig_file_id` VARCHAR(64) NOT NULL,
+            `path` VARCHAR(12288) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `num_uncompressed_bytes` BIGINT NOT NULL,
+            `begin_message_ix` BIGINT NOT NULL,
+            `num_messages` BIGINT NOT NULL,
+            `archive_id` VARCHAR(64) NOT NULL,
+            KEY `files_path` (path(768)) USING BTREE,
+            KEY `files_archive_id` (`archive_id`) USING BTREE,
+            PRIMARY KEY (`id`)
+        ) ROW_FORMAT=DYNAMIC
+        """
+    )
+
+
+def _create_column_metadata_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{COLUMN_METADATA_TABLE_SUFFIX}` (
+            `name` VARCHAR(512) NOT NULL,
+            `type` TINYINT NOT NULL,
+            PRIMARY KEY (`name`, `type`)
+        )
+        """
+    )
+
+
+def create_datasets_table(db_cursor, table_prefix: str) -> None:
+    """
+    Creates the table to store the datasets information (for clp-s only).
path: components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py
user: login: kirkrodrigues
body: We try to include an empty line between the description and parameters.
created_at: 2025-04-23T02:04:29Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,125 @@
+import typing
+
+from clp_py_utils.clp_config import (
+    ARCHIVE_TAGS_TABLE_SUFFIX,
+    ARCHIVES_TABLE_SUFFIX,
+    COLUMN_METADATA_TABLE_SUFFIX,
+    DATASETS_TABLE_SUFFIX,
+    FILES_TABLE_SUFFIX,
+    TAGS_TABLE_SUFFIX,
+)
+
+
+def _create_archives_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{ARCHIVES_TABLE_SUFFIX}` (
+            `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
+            `id` VARCHAR(64) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `uncompressed_size` BIGINT NOT NULL,
+            `size` BIGINT NOT NULL,
+            `creator_id` VARCHAR(64) NOT NULL,
+            `creation_ix` INT NOT NULL,
+            KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
+            UNIQUE KEY `archive_id` (`id`) USING BTREE,
+            PRIMARY KEY (`pagination_id`)
+        )
+        """
+    )
+
+
+def _create_tags_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{TAGS_TABLE_SUFFIX}` (
+            `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
+            `tag_name` VARCHAR(255) NOT NULL,
+            UNIQUE KEY (`tag_name`) USING BTREE,
+            PRIMARY KEY (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_archive_tags_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{ARCHIVE_TAGS_TABLE_SUFFIX}` (
+            `archive_id` VARCHAR(64) NOT NULL,
+            `tag_id` INT unsigned NOT NULL,
+            PRIMARY KEY (`archive_id`,`tag_id`),
+            FOREIGN KEY (`archive_id`) REFERENCES `{table_prefix}{ARCHIVES_TABLE_SUFFIX}` (`id`),
+            FOREIGN KEY (`tag_id`) REFERENCES `{table_prefix}{TAGS_TABLE_SUFFIX}` (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_files_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{FILES_TABLE_SUFFIX}` (
+            `id` VARCHAR(64) NOT NULL,
+            `orig_file_id` VARCHAR(64) NOT NULL,
+            `path` VARCHAR(12288) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `num_uncompressed_bytes` BIGINT NOT NULL,
+            `begin_message_ix` BIGINT NOT NULL,
+            `num_messages` BIGINT NOT NULL,
+            `archive_id` VARCHAR(64) NOT NULL,
+            KEY `files_path` (path(768)) USING BTREE,
+            KEY `files_archive_id` (`archive_id`) USING BTREE,
+            PRIMARY KEY (`id`)
+        ) ROW_FORMAT=DYNAMIC
+        """
+    )
+
+
+def _create_column_metadata_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{COLUMN_METADATA_TABLE_SUFFIX}` (
+            `name` VARCHAR(512) NOT NULL,
+            `type` TINYINT NOT NULL,
+            PRIMARY KEY (`name`, `type`)
+        )
+        """
+    )
+
+
+def create_datasets_table(db_cursor, table_prefix: str) -> None:
+    """
+    Creates the table to store the datasets information (for clp-s only).
+    :param db_cursor: The database cursor to execute the table creation.
+    :param table_prefix: A string prefix applied to the dataset table name.
+    """
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{DATASETS_TABLE_SUFFIX}` (
+            `name` VARCHAR(255) NOT NULL,
+            `archive_storage_directory` VARCHAR(4096) NOT NULL,
+            PRIMARY KEY (`name`)
+        )
+        """
+    )
+
+
+def create_metadata_db_tables(db_cursor, table_prefix: str, dataset: str | None = None) -> None:
+    """
+    Creates the standard set of SQL tables in the metadata database for recording compression tasks'
+    results.
path: components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py
user: login: kirkrodrigues
body: ```suggestion
    Creates the standard set of tables for CLP's metadata.

```
created_at: 2025-04-23T02:04:58Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,125 @@
+import typing
+
+from clp_py_utils.clp_config import (
+    ARCHIVE_TAGS_TABLE_SUFFIX,
+    ARCHIVES_TABLE_SUFFIX,
+    COLUMN_METADATA_TABLE_SUFFIX,
+    DATASETS_TABLE_SUFFIX,
+    FILES_TABLE_SUFFIX,
+    TAGS_TABLE_SUFFIX,
+)
+
+
+def _create_archives_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{ARCHIVES_TABLE_SUFFIX}` (
+            `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
+            `id` VARCHAR(64) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `uncompressed_size` BIGINT NOT NULL,
+            `size` BIGINT NOT NULL,
+            `creator_id` VARCHAR(64) NOT NULL,
+            `creation_ix` INT NOT NULL,
+            KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
+            UNIQUE KEY `archive_id` (`id`) USING BTREE,
+            PRIMARY KEY (`pagination_id`)
+        )
+        """
+    )
+
+
+def _create_tags_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{TAGS_TABLE_SUFFIX}` (
+            `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
+            `tag_name` VARCHAR(255) NOT NULL,
+            UNIQUE KEY (`tag_name`) USING BTREE,
+            PRIMARY KEY (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_archive_tags_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{ARCHIVE_TAGS_TABLE_SUFFIX}` (
+            `archive_id` VARCHAR(64) NOT NULL,
+            `tag_id` INT unsigned NOT NULL,
+            PRIMARY KEY (`archive_id`,`tag_id`),
+            FOREIGN KEY (`archive_id`) REFERENCES `{table_prefix}{ARCHIVES_TABLE_SUFFIX}` (`id`),
+            FOREIGN KEY (`tag_id`) REFERENCES `{table_prefix}{TAGS_TABLE_SUFFIX}` (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_files_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{FILES_TABLE_SUFFIX}` (
+            `id` VARCHAR(64) NOT NULL,
+            `orig_file_id` VARCHAR(64) NOT NULL,
+            `path` VARCHAR(12288) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `num_uncompressed_bytes` BIGINT NOT NULL,
+            `begin_message_ix` BIGINT NOT NULL,
+            `num_messages` BIGINT NOT NULL,
+            `archive_id` VARCHAR(64) NOT NULL,
+            KEY `files_path` (path(768)) USING BTREE,
+            KEY `files_archive_id` (`archive_id`) USING BTREE,
+            PRIMARY KEY (`id`)
+        ) ROW_FORMAT=DYNAMIC
+        """
+    )
+
+
+def _create_column_metadata_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{COLUMN_METADATA_TABLE_SUFFIX}` (
+            `name` VARCHAR(512) NOT NULL,
+            `type` TINYINT NOT NULL,
+            PRIMARY KEY (`name`, `type`)
+        )
+        """
+    )
+
+
+def create_datasets_table(db_cursor, table_prefix: str) -> None:
+    """
+    Creates the table to store the datasets information (for clp-s only).
+    :param db_cursor: The database cursor to execute the table creation.
+    :param table_prefix: A string prefix applied to the dataset table name.
+    """
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{DATASETS_TABLE_SUFFIX}` (
+            `name` VARCHAR(255) NOT NULL,
+            `archive_storage_directory` VARCHAR(4096) NOT NULL,
+            PRIMARY KEY (`name`)
+        )
+        """
+    )
+
+
+def create_metadata_db_tables(db_cursor, table_prefix: str, dataset: str | None = None) -> None:
+    """
+    Creates the standard set of SQL tables in the metadata database for recording compression tasks'
+    results.
+    :param db_cursor: The database cursor to execute the table creations.
+    :param table_prefix: A string prefix applied to all table names.
+    :param dataset: If provided, dataset-specific set of tables are created (for clp-s only).
path: components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py
user: login: kirkrodrigues
body: ```suggestion
    :param table_prefix: A string to prepend to all table names.
    :param dataset: If set, all tables will be named in a dataset-specific manner and the
    column-metadata table will be created.
```
created_at: 2025-04-23T02:05:15Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,125 @@
+import typing
+
+from clp_py_utils.clp_config import (
+    ARCHIVE_TAGS_TABLE_SUFFIX,
+    ARCHIVES_TABLE_SUFFIX,
+    COLUMN_METADATA_TABLE_SUFFIX,
+    DATASETS_TABLE_SUFFIX,
+    FILES_TABLE_SUFFIX,
+    TAGS_TABLE_SUFFIX,
+)
+
+
+def _create_archives_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{ARCHIVES_TABLE_SUFFIX}` (
+            `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
+            `id` VARCHAR(64) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `uncompressed_size` BIGINT NOT NULL,
+            `size` BIGINT NOT NULL,
+            `creator_id` VARCHAR(64) NOT NULL,
+            `creation_ix` INT NOT NULL,
+            KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
+            UNIQUE KEY `archive_id` (`id`) USING BTREE,
+            PRIMARY KEY (`pagination_id`)
+        )
+        """
+    )
+
+
+def _create_tags_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{TAGS_TABLE_SUFFIX}` (
+            `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
+            `tag_name` VARCHAR(255) NOT NULL,
+            UNIQUE KEY (`tag_name`) USING BTREE,
+            PRIMARY KEY (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_archive_tags_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{ARCHIVE_TAGS_TABLE_SUFFIX}` (
+            `archive_id` VARCHAR(64) NOT NULL,
+            `tag_id` INT unsigned NOT NULL,
+            PRIMARY KEY (`archive_id`,`tag_id`),
+            FOREIGN KEY (`archive_id`) REFERENCES `{table_prefix}{ARCHIVES_TABLE_SUFFIX}` (`id`),
+            FOREIGN KEY (`tag_id`) REFERENCES `{table_prefix}{TAGS_TABLE_SUFFIX}` (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_files_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{FILES_TABLE_SUFFIX}` (
+            `id` VARCHAR(64) NOT NULL,
+            `orig_file_id` VARCHAR(64) NOT NULL,
+            `path` VARCHAR(12288) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `num_uncompressed_bytes` BIGINT NOT NULL,
+            `begin_message_ix` BIGINT NOT NULL,
+            `num_messages` BIGINT NOT NULL,
+            `archive_id` VARCHAR(64) NOT NULL,
+            KEY `files_path` (path(768)) USING BTREE,
+            KEY `files_archive_id` (`archive_id`) USING BTREE,
+            PRIMARY KEY (`id`)
+        ) ROW_FORMAT=DYNAMIC
+        """
+    )
+
+
+def _create_column_metadata_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{COLUMN_METADATA_TABLE_SUFFIX}` (
+            `name` VARCHAR(512) NOT NULL,
+            `type` TINYINT NOT NULL,
+            PRIMARY KEY (`name`, `type`)
+        )
+        """
+    )
+
+
+def create_datasets_table(db_cursor, table_prefix: str) -> None:
+    """
+    Creates the table to store the datasets information (for clp-s only).
+    :param db_cursor: The database cursor to execute the table creation.
+    :param table_prefix: A string prefix applied to the dataset table name.
+    """
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{DATASETS_TABLE_SUFFIX}` (
+            `name` VARCHAR(255) NOT NULL,
+            `archive_storage_directory` VARCHAR(4096) NOT NULL,
+            PRIMARY KEY (`name`)
+        )
+        """
+    )
+
+
+def create_metadata_db_tables(db_cursor, table_prefix: str, dataset: str | None = None) -> None:
+    """
+    Creates the standard set of SQL tables in the metadata database for recording compression tasks'
+    results.
+    :param db_cursor: The database cursor to execute the table creations.
+    :param table_prefix: A string prefix applied to all table names.
+    :param dataset: If provided, dataset-specific set of tables are created (for clp-s only).
+    """
+    if dataset is not None:
+        table_prefix = f"{table_prefix}{dataset}_"
+        _create_column_metadata_table(db_cursor, table_prefix)
+
+    _create_archives_table(db_cursor, table_prefix)
+    _create_tags_table(db_cursor, table_prefix)
+    _create_archive_tags_table(db_cursor, table_prefix)
path: components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py
user: login: kirkrodrigues
body: Sorry, this is my mistake---for these 3 tables, the way you had it before where you set the names outside and then passed them in was more sensible.
created_at: 2025-04-23T02:06:16Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,125 @@
+import typing
+
+from clp_py_utils.clp_config import (
+    ARCHIVE_TAGS_TABLE_SUFFIX,
+    ARCHIVES_TABLE_SUFFIX,
+    COLUMN_METADATA_TABLE_SUFFIX,
+    DATASETS_TABLE_SUFFIX,
+    FILES_TABLE_SUFFIX,
+    TAGS_TABLE_SUFFIX,
+)
+
+
+def _create_archives_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{ARCHIVES_TABLE_SUFFIX}` (
+            `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
+            `id` VARCHAR(64) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `uncompressed_size` BIGINT NOT NULL,
+            `size` BIGINT NOT NULL,
+            `creator_id` VARCHAR(64) NOT NULL,
+            `creation_ix` INT NOT NULL,
+            KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
+            UNIQUE KEY `archive_id` (`id`) USING BTREE,
+            PRIMARY KEY (`pagination_id`)
+        )
+        """
+    )
+
+
+def _create_tags_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{TAGS_TABLE_SUFFIX}` (
+            `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
+            `tag_name` VARCHAR(255) NOT NULL,
+            UNIQUE KEY (`tag_name`) USING BTREE,
+            PRIMARY KEY (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_archive_tags_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{ARCHIVE_TAGS_TABLE_SUFFIX}` (
+            `archive_id` VARCHAR(64) NOT NULL,
+            `tag_id` INT unsigned NOT NULL,
+            PRIMARY KEY (`archive_id`,`tag_id`),
+            FOREIGN KEY (`archive_id`) REFERENCES `{table_prefix}{ARCHIVES_TABLE_SUFFIX}` (`id`),
+            FOREIGN KEY (`tag_id`) REFERENCES `{table_prefix}{TAGS_TABLE_SUFFIX}` (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_files_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{FILES_TABLE_SUFFIX}` (
+            `id` VARCHAR(64) NOT NULL,
+            `orig_file_id` VARCHAR(64) NOT NULL,
+            `path` VARCHAR(12288) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `num_uncompressed_bytes` BIGINT NOT NULL,
+            `begin_message_ix` BIGINT NOT NULL,
+            `num_messages` BIGINT NOT NULL,
+            `archive_id` VARCHAR(64) NOT NULL,
+            KEY `files_path` (path(768)) USING BTREE,
+            KEY `files_archive_id` (`archive_id`) USING BTREE,
+            PRIMARY KEY (`id`)
+        ) ROW_FORMAT=DYNAMIC
+        """
+    )
+
+
+def _create_column_metadata_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{COLUMN_METADATA_TABLE_SUFFIX}` (
+            `name` VARCHAR(512) NOT NULL,
+            `type` TINYINT NOT NULL,
+            PRIMARY KEY (`name`, `type`)
+        )
+        """
+    )
+
+
+def create_datasets_table(db_cursor, table_prefix: str) -> None:
+    """
+    Creates the table to store the datasets information (for clp-s only).
+    :param db_cursor: The database cursor to execute the table creation.
+    :param table_prefix: A string prefix applied to the dataset table name.
+    """
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{DATASETS_TABLE_SUFFIX}` (
+            `name` VARCHAR(255) NOT NULL,
+            `archive_storage_directory` VARCHAR(4096) NOT NULL,
+            PRIMARY KEY (`name`)
+        )
+        """
+    )
+
+
+def create_metadata_db_tables(db_cursor, table_prefix: str, dataset: str | None = None) -> None:
path: components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py
user: login: kirkrodrigues
body: To support Python 3.8 (see pyproject.toml), the `|` syntax requires `from __future__ import annotations`
created_at: 2025-04-23T02:07:27Z
author_association: MEMBER
diff_hunk: @@ -1,12 +1,21 @@
 #!/usr/bin/env python3
 import argparse
 import logging
+import pathlib
path: components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py
user: login: kirkrodrigues
body: Can we use `from pathlib import Path` and then use `Path` below?
created_at: 2025-04-23T02:08:20Z
author_association: MEMBER
diff_hunk: @@ -23,88 +32,36 @@
 def main(argv):
     args_parser = argparse.ArgumentParser(description="Sets up CLP's metadata tables.")
     args_parser.add_argument("--config", required=True, help="Database config file.")
+    args_parser.add_argument(
+        "--storage-engine",
+        type=str,
+        choices=[engine.value for engine in StorageEngine],
+        required=True,
+        help="Compression storage engine to use.",
path: components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py
user: login: kirkrodrigues
body: ```suggestion
        help="Storage engine to create tables for.",
```
created_at: 2025-04-23T02:08:32Z
author_association: MEMBER
diff_hunk: @@ -23,88 +32,36 @@
 def main(argv):
     args_parser = argparse.ArgumentParser(description="Sets up CLP's metadata tables.")
     args_parser.add_argument("--config", required=True, help="Database config file.")
+    args_parser.add_argument(
+        "--storage-engine",
+        type=str,
+        choices=[engine.value for engine in StorageEngine],
+        required=True,
+        help="Compression storage engine to use.",
+    )
     parsed_args = args_parser.parse_args(argv[1:])
 
+    config_file_path = pathlib.Path(parsed_args.config)
+    storage_engine = StorageEngine(parsed_args.storage_engine)
+
     try:
-        database_config = Database.parse_obj(read_yaml_config_file(parsed_args.config))
+        database_config = Database.parse_obj(read_yaml_config_file(config_file_path))
         if database_config is None:
-            raise ValueError(f"Database configuration file '{parsed_args.config}' is empty.")
+            raise ValueError(f"Database configuration file '{config_file_path}' is empty.")
         sql_adapter = SQL_Adapter(database_config)
         clp_db_connection_params = database_config.get_clp_connection_params_and_type(True)
         table_prefix = clp_db_connection_params["table_prefix"]
         with closing(sql_adapter.create_connection(True)) as metadata_db, closing(
             metadata_db.cursor(dictionary=True)
         ) as metadata_db_cursor:
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}archives` (
-                    `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
-                    `id` VARCHAR(64) NOT NULL,
-                    `begin_timestamp` BIGINT NOT NULL,
-                    `end_timestamp` BIGINT NOT NULL,
-                    `uncompressed_size` BIGINT NOT NULL,
-                    `size` BIGINT NOT NULL,
-                    `creator_id` VARCHAR(64) NOT NULL,
-                    `creation_ix` INT NOT NULL,
-                    KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
-                    UNIQUE KEY `archive_id` (`id`) USING BTREE,
-                    PRIMARY KEY (`pagination_id`)
-                )
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}tags` (
-                    `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
-                    `tag_name` VARCHAR(255) NOT NULL,
-                    UNIQUE KEY (`tag_name`) USING BTREE,
-                    PRIMARY KEY (`tag_id`)
-                )
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}archive_tags` (
-                    `archive_id` VARCHAR(64) NOT NULL,
-                    `tag_id` INT unsigned NOT NULL,
-                    PRIMARY KEY (`archive_id`,`tag_id`),
-                    FOREIGN KEY (`archive_id`) REFERENCES `{table_prefix}archives` (`id`),
-                    FOREIGN KEY (`tag_id`) REFERENCES `{table_prefix}tags` (`tag_id`)
+            if StorageEngine.CLP_S == storage_engine:
+                create_metadata_db_tables(
+                    metadata_db_cursor, table_prefix, dataset=CLP_DEFAULT_DATASET_NAME
                 )
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}files` (
-                    `id` VARCHAR(64) NOT NULL,
-                    `orig_file_id` VARCHAR(64) NOT NULL,
-                    `path` VARCHAR(12288) NOT NULL,
-                    `begin_timestamp` BIGINT NOT NULL,
-                    `end_timestamp` BIGINT NOT NULL,
-                    `num_uncompressed_bytes` BIGINT NOT NULL,
-                    `begin_message_ix` BIGINT NOT NULL,
-                    `num_messages` BIGINT NOT NULL,
-                    `archive_id` VARCHAR(64) NOT NULL,
-                    KEY `files_path` (path(768)) USING BTREE,
-                    KEY `files_archive_id` (`archive_id`) USING BTREE,
-                    PRIMARY KEY (`id`)
-                ) ROW_FORMAT=DYNAMIC
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}column_metadata_default` (
-                    `name` VARCHAR(512) NOT NULL,
-                    `type` TINYINT NOT NULL,
-                    PRIMARY KEY (`name`, `type`)
-                )
-                """
-            )
-
+                # TODO: populate the dataset table with proper default name and default path.
+                create_datasets_table(metadata_db_cursor, table_prefix)
+            # TODO: Remove the default db tables for CLP_S after the dataset feature is implemented.
path: components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py
user: login: kirkrodrigues
body: There are a few issues (one I should've brought up in the previous review, sorry):

* `indexer` (see compression_task.py) is using the table name `clp_column_metadata_default` rather than `clp_{dataset}_column_metadata`, so we'll need to update `indexer` to use the correct table name.
* Can we defer creating the dataset-specific tables until the next PR? I know they're harmless but logically, it makes more sense to create them when we actually have the dataset feature implemented (and it'll lead to less confusion if someone is debugging the package before the next PR).
* To ensure the column_metadata table is created even though we're not passing in a dataset name, we can make its creation unconditional (like it was before this PR); and we can name it `clp_column_metadata`.
created_at: 2025-04-23T02:23:57Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,140 @@
+from __future__ import annotations
+
+from clp_py_utils.clp_config import (
+    ARCHIVE_TAGS_TABLE_SUFFIX,
+    ARCHIVES_TABLE_SUFFIX,
+    CLP_DEFAULT_DATASET_NAME,
+    COLUMN_METADATA_TABLE_SUFFIX,
+    DATASETS_TABLE_SUFFIX,
+    FILES_TABLE_SUFFIX,
+    TAGS_TABLE_SUFFIX,
+)
+
+
+def _create_archives_table(db_cursor, archives_table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{archives_table_name}` (
+            `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
+            `id` VARCHAR(64) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `uncompressed_size` BIGINT NOT NULL,
+            `size` BIGINT NOT NULL,
+            `creator_id` VARCHAR(64) NOT NULL,
+            `creation_ix` INT NOT NULL,
+            KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
+            UNIQUE KEY `archive_id` (`id`) USING BTREE,
+            PRIMARY KEY (`pagination_id`)
+        )
+        """
+    )
+
+
+def _create_tags_table(db_cursor, tags_table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{tags_table_name}` (
+            `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
+            `tag_name` VARCHAR(255) NOT NULL,
+            UNIQUE KEY (`tag_name`) USING BTREE,
+            PRIMARY KEY (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_archive_tags_table(
+    db_cursor, archive_tags_table_name: str, archives_table_name: str, tags_table_name: str
+) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{archive_tags_table_name}` (
+            `archive_id` VARCHAR(64) NOT NULL,
+            `tag_id` INT unsigned NOT NULL,
+            PRIMARY KEY (`archive_id`,`tag_id`),
+            FOREIGN KEY (`archive_id`) REFERENCES `{archives_table_name}` (`id`),
+            FOREIGN KEY (`tag_id`) REFERENCES `{tags_table_name}` (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_files_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{FILES_TABLE_SUFFIX}` (
+            `id` VARCHAR(64) NOT NULL,
+            `orig_file_id` VARCHAR(64) NOT NULL,
+            `path` VARCHAR(12288) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `num_uncompressed_bytes` BIGINT NOT NULL,
+            `begin_message_ix` BIGINT NOT NULL,
+            `num_messages` BIGINT NOT NULL,
+            `archive_id` VARCHAR(64) NOT NULL,
+            KEY `files_path` (path(768)) USING BTREE,
+            KEY `files_archive_id` (`archive_id`) USING BTREE,
+            PRIMARY KEY (`id`)
+        ) ROW_FORMAT=DYNAMIC
+        """
+    )
+
+
+def _create_column_metadata_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `name` VARCHAR(512) NOT NULL,
+            `type` TINYINT NOT NULL,
+            PRIMARY KEY (`name`, `type`)
+        )
+        """
+    )
+
+
+def create_datasets_table(db_cursor, table_prefix: str) -> None:
+    """
+    Creates the dataset information table.
+
+    :param db_cursor: The database cursor to execute the table creation.
+    :param table_prefix: A string to prepend to the table name.
+    """
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{DATASETS_TABLE_SUFFIX}` (
+            `name` VARCHAR(255) NOT NULL,
+            `archive_storage_directory` VARCHAR(4096) NOT NULL,
+            PRIMARY KEY (`name`)
+        )
+        """
+    )
+
+
+def create_metadata_db_tables(db_cursor, table_prefix: str, dataset: str | None = None) -> None:
+    """
+    Creates the standard set of tables for CLP's metadata.
+
+    :param db_cursor: The database cursor to execute the table creations.
+    :param table_prefix: A string to prepend to all table names.
+    :param dataset: If set, all tables will be named in a dataset-specific manner and the
+    column-metadata table will be created.
path: components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py
user: login: kirkrodrigues
body: ```suggestion
    :param dataset: If set, all tables will be named in a dataset-specific manner.
```
created_at: 2025-04-23T17:25:19Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,140 @@
+from __future__ import annotations
+
+from clp_py_utils.clp_config import (
+    ARCHIVE_TAGS_TABLE_SUFFIX,
+    ARCHIVES_TABLE_SUFFIX,
+    CLP_DEFAULT_DATASET_NAME,
+    COLUMN_METADATA_TABLE_SUFFIX,
+    DATASETS_TABLE_SUFFIX,
+    FILES_TABLE_SUFFIX,
+    TAGS_TABLE_SUFFIX,
+)
+
+
+def _create_archives_table(db_cursor, archives_table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{archives_table_name}` (
+            `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
+            `id` VARCHAR(64) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `uncompressed_size` BIGINT NOT NULL,
+            `size` BIGINT NOT NULL,
+            `creator_id` VARCHAR(64) NOT NULL,
+            `creation_ix` INT NOT NULL,
+            KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
+            UNIQUE KEY `archive_id` (`id`) USING BTREE,
+            PRIMARY KEY (`pagination_id`)
+        )
+        """
+    )
+
+
+def _create_tags_table(db_cursor, tags_table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{tags_table_name}` (
+            `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
+            `tag_name` VARCHAR(255) NOT NULL,
+            UNIQUE KEY (`tag_name`) USING BTREE,
+            PRIMARY KEY (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_archive_tags_table(
+    db_cursor, archive_tags_table_name: str, archives_table_name: str, tags_table_name: str
+) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{archive_tags_table_name}` (
+            `archive_id` VARCHAR(64) NOT NULL,
+            `tag_id` INT unsigned NOT NULL,
+            PRIMARY KEY (`archive_id`,`tag_id`),
+            FOREIGN KEY (`archive_id`) REFERENCES `{archives_table_name}` (`id`),
+            FOREIGN KEY (`tag_id`) REFERENCES `{tags_table_name}` (`tag_id`)
+        )
+        """
+    )
+
+
+def _create_files_table(db_cursor, table_prefix: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{FILES_TABLE_SUFFIX}` (
+            `id` VARCHAR(64) NOT NULL,
+            `orig_file_id` VARCHAR(64) NOT NULL,
+            `path` VARCHAR(12288) NOT NULL,
+            `begin_timestamp` BIGINT NOT NULL,
+            `end_timestamp` BIGINT NOT NULL,
+            `num_uncompressed_bytes` BIGINT NOT NULL,
+            `begin_message_ix` BIGINT NOT NULL,
+            `num_messages` BIGINT NOT NULL,
+            `archive_id` VARCHAR(64) NOT NULL,
+            KEY `files_path` (path(768)) USING BTREE,
+            KEY `files_archive_id` (`archive_id`) USING BTREE,
+            PRIMARY KEY (`id`)
+        ) ROW_FORMAT=DYNAMIC
+        """
+    )
+
+
+def _create_column_metadata_table(db_cursor, table_name: str) -> None:
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_name}` (
+            `name` VARCHAR(512) NOT NULL,
+            `type` TINYINT NOT NULL,
+            PRIMARY KEY (`name`, `type`)
+        )
+        """
+    )
+
+
+def create_datasets_table(db_cursor, table_prefix: str) -> None:
+    """
+    Creates the dataset information table.
+
+    :param db_cursor: The database cursor to execute the table creation.
+    :param table_prefix: A string to prepend to the table name.
+    """
+    db_cursor.execute(
+        f"""
+        CREATE TABLE IF NOT EXISTS `{table_prefix}{DATASETS_TABLE_SUFFIX}` (
+            `name` VARCHAR(255) NOT NULL,
+            `archive_storage_directory` VARCHAR(4096) NOT NULL,
+            PRIMARY KEY (`name`)
+        )
+        """
+    )
+
+
+def create_metadata_db_tables(db_cursor, table_prefix: str, dataset: str | None = None) -> None:
+    """
+    Creates the standard set of tables for CLP's metadata.
+
+    :param db_cursor: The database cursor to execute the table creations.
+    :param table_prefix: A string to prepend to all table names.
+    :param dataset: If set, all tables will be named in a dataset-specific manner and the
+    column-metadata table will be created.
+    """
+    if dataset is not None:
+        table_prefix = f"{table_prefix}{dataset}_"
+
+    archives_table_name = f"{table_prefix}{ARCHIVES_TABLE_SUFFIX}"
+    tags_table_name = f"{table_prefix}{TAGS_TABLE_SUFFIX}"
+    archive_tags_table_name = f"{table_prefix}{ARCHIVE_TAGS_TABLE_SUFFIX}"
+    # TODO: Update the default column metadata table name after updating the indexer.
path: components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py
user: login: kirkrodrigues
body: ```suggestion

    # TODO: Update this to
    # {table_prefix}{CLP_DEFAULT_DATASET_NAME}_{COLUMN_METADATA_TABLE_SUFFIX} when we can also
    # change the indexer to match.
```

* We try to include an empty line before comment blocks since it makes it a bit easier to read.
* I tried to make the TODO more actionable in case it's not you who addresses it.
created_at: 2025-04-23T17:26:06Z
author_association: MEMBER
diff_hunk: @@ -23,88 +31,36 @@
 def main(argv):
     args_parser = argparse.ArgumentParser(description="Sets up CLP's metadata tables.")
     args_parser.add_argument("--config", required=True, help="Database config file.")
+    args_parser.add_argument(
+        "--storage-engine",
+        type=str,
+        choices=[engine.value for engine in StorageEngine],
+        required=True,
+        help="Storage engine to create tables for.",
+    )
     parsed_args = args_parser.parse_args(argv[1:])
 
+    config_file_path = Path(parsed_args.config)
+    storage_engine = StorageEngine(parsed_args.storage_engine)
+
     try:
-        database_config = Database.parse_obj(read_yaml_config_file(parsed_args.config))
+        database_config = Database.parse_obj(read_yaml_config_file(config_file_path))
         if database_config is None:
-            raise ValueError(f"Database configuration file '{parsed_args.config}' is empty.")
+            raise ValueError(f"Database configuration file '{config_file_path}' is empty.")
         sql_adapter = SQL_Adapter(database_config)
         clp_db_connection_params = database_config.get_clp_connection_params_and_type(True)
         table_prefix = clp_db_connection_params["table_prefix"]
         with closing(sql_adapter.create_connection(True)) as metadata_db, closing(
             metadata_db.cursor(dictionary=True)
         ) as metadata_db_cursor:
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}archives` (
-                    `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
-                    `id` VARCHAR(64) NOT NULL,
-                    `begin_timestamp` BIGINT NOT NULL,
-                    `end_timestamp` BIGINT NOT NULL,
-                    `uncompressed_size` BIGINT NOT NULL,
-                    `size` BIGINT NOT NULL,
-                    `creator_id` VARCHAR(64) NOT NULL,
-                    `creation_ix` INT NOT NULL,
-                    KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
-                    UNIQUE KEY `archive_id` (`id`) USING BTREE,
-                    PRIMARY KEY (`pagination_id`)
-                )
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}tags` (
-                    `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
-                    `tag_name` VARCHAR(255) NOT NULL,
-                    UNIQUE KEY (`tag_name`) USING BTREE,
-                    PRIMARY KEY (`tag_id`)
-                )
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}archive_tags` (
-                    `archive_id` VARCHAR(64) NOT NULL,
-                    `tag_id` INT unsigned NOT NULL,
-                    PRIMARY KEY (`archive_id`,`tag_id`),
-                    FOREIGN KEY (`archive_id`) REFERENCES `{table_prefix}archives` (`id`),
-                    FOREIGN KEY (`tag_id`) REFERENCES `{table_prefix}tags` (`tag_id`)
-                )
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}files` (
-                    `id` VARCHAR(64) NOT NULL,
-                    `orig_file_id` VARCHAR(64) NOT NULL,
-                    `path` VARCHAR(12288) NOT NULL,
-                    `begin_timestamp` BIGINT NOT NULL,
-                    `end_timestamp` BIGINT NOT NULL,
-                    `num_uncompressed_bytes` BIGINT NOT NULL,
-                    `begin_message_ix` BIGINT NOT NULL,
-                    `num_messages` BIGINT NOT NULL,
-                    `archive_id` VARCHAR(64) NOT NULL,
-                    KEY `files_path` (path(768)) USING BTREE,
-                    KEY `files_archive_id` (`archive_id`) USING BTREE,
-                    PRIMARY KEY (`id`)
-                ) ROW_FORMAT=DYNAMIC
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}column_metadata_default` (
-                    `name` VARCHAR(512) NOT NULL,
-                    `type` TINYINT NOT NULL,
-                    PRIMARY KEY (`name`, `type`)
-                )
-                """
-            )
-
+            # TODO: After the dataset feature is implemented:
+            #       1. Populate the `clp-datasets` table with the name and path for the `default`
+            #          dataset.
+            #       2. Create a set of metadata tables for the `default` dataset.
+            #       3. Remove the default db tables for CLP_S.
path: components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py
user: login: kirkrodrigues
body: ```suggestion
            # TODO: After the dataset feature is fully implemented, for clp-json:
            # 1. Populate the datasets table with the name and path for the "default" dataset.
            # 2. Change the metadata tables to be specific to the "default" dataset.
```

I think that's semantically equivalent?
created_at: 2025-04-23T17:26:59Z
author_association: MEMBER
diff_hunk: @@ -23,88 +31,36 @@
 def main(argv):
     args_parser = argparse.ArgumentParser(description="Sets up CLP's metadata tables.")
     args_parser.add_argument("--config", required=True, help="Database config file.")
+    args_parser.add_argument(
+        "--storage-engine",
+        type=str,
+        choices=[engine.value for engine in StorageEngine],
+        required=True,
+        help="Storage engine to create tables for.",
+    )
     parsed_args = args_parser.parse_args(argv[1:])
 
+    config_file_path = Path(parsed_args.config)
+    storage_engine = StorageEngine(parsed_args.storage_engine)
+
     try:
-        database_config = Database.parse_obj(read_yaml_config_file(parsed_args.config))
+        database_config = Database.parse_obj(read_yaml_config_file(config_file_path))
         if database_config is None:
-            raise ValueError(f"Database configuration file '{parsed_args.config}' is empty.")
+            raise ValueError(f"Database configuration file '{config_file_path}' is empty.")
         sql_adapter = SQL_Adapter(database_config)
         clp_db_connection_params = database_config.get_clp_connection_params_and_type(True)
         table_prefix = clp_db_connection_params["table_prefix"]
         with closing(sql_adapter.create_connection(True)) as metadata_db, closing(
             metadata_db.cursor(dictionary=True)
         ) as metadata_db_cursor:
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}archives` (
-                    `pagination_id` BIGINT unsigned NOT NULL AUTO_INCREMENT,
-                    `id` VARCHAR(64) NOT NULL,
-                    `begin_timestamp` BIGINT NOT NULL,
-                    `end_timestamp` BIGINT NOT NULL,
-                    `uncompressed_size` BIGINT NOT NULL,
-                    `size` BIGINT NOT NULL,
-                    `creator_id` VARCHAR(64) NOT NULL,
-                    `creation_ix` INT NOT NULL,
-                    KEY `archives_creation_order` (`creator_id`,`creation_ix`) USING BTREE,
-                    UNIQUE KEY `archive_id` (`id`) USING BTREE,
-                    PRIMARY KEY (`pagination_id`)
-                )
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}tags` (
-                    `tag_id` INT unsigned NOT NULL AUTO_INCREMENT,
-                    `tag_name` VARCHAR(255) NOT NULL,
-                    UNIQUE KEY (`tag_name`) USING BTREE,
-                    PRIMARY KEY (`tag_id`)
-                )
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}archive_tags` (
-                    `archive_id` VARCHAR(64) NOT NULL,
-                    `tag_id` INT unsigned NOT NULL,
-                    PRIMARY KEY (`archive_id`,`tag_id`),
-                    FOREIGN KEY (`archive_id`) REFERENCES `{table_prefix}archives` (`id`),
-                    FOREIGN KEY (`tag_id`) REFERENCES `{table_prefix}tags` (`tag_id`)
-                )
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}files` (
-                    `id` VARCHAR(64) NOT NULL,
-                    `orig_file_id` VARCHAR(64) NOT NULL,
-                    `path` VARCHAR(12288) NOT NULL,
-                    `begin_timestamp` BIGINT NOT NULL,
-                    `end_timestamp` BIGINT NOT NULL,
-                    `num_uncompressed_bytes` BIGINT NOT NULL,
-                    `begin_message_ix` BIGINT NOT NULL,
-                    `num_messages` BIGINT NOT NULL,
-                    `archive_id` VARCHAR(64) NOT NULL,
-                    KEY `files_path` (path(768)) USING BTREE,
-                    KEY `files_archive_id` (`archive_id`) USING BTREE,
-                    PRIMARY KEY (`id`)
-                ) ROW_FORMAT=DYNAMIC
-                """
-            )
-
-            metadata_db_cursor.execute(
-                f"""
-                CREATE TABLE IF NOT EXISTS `{table_prefix}column_metadata_default` (
-                    `name` VARCHAR(512) NOT NULL,
-                    `type` TINYINT NOT NULL,
-                    PRIMARY KEY (`name`, `type`)
-                )
-                """
-            )
-
+            # TODO: After the dataset feature is implemented:
+            #       1. Populate the `clp-datasets` table with the name and path for the `default`
+            #          dataset.
+            #       2. Create a set of metadata tables for the `default` dataset.
+            #       3. Remove the default db tables for CLP_S.
path: components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py
user: login: Bill-hbrhbr
body: yeah
created_at: 2025-04-23T18:10:20Z
author_association: CONTRIBUTOR

PR comments:
user: login: coderabbitai[bot]
created_at: 2025-04-17T07:41:59Z
body: <!-- This is an auto-generated comment: summarize by coderabbit.ai -->
<!-- walkthrough_start -->

## Walkthrough

This update introduces a new module containing utility functions for creating metadata tables in a MariaDB or MySQL database. Several string constants representing table suffixes are added to the configuration file. The table creation logic in an initialization script is refactored to use a new consolidated function from the utility module, replacing multiple explicit SQL statements with a single function call. The new module defines individual functions for creating each table and a coordinating function to create all required tables with optional dataset-specific behaviour. Additionally, command-line argument handling is enhanced in related scripts to include a required `--storage-engine` parameter, which influences table creation behaviour.

## Changes

| File(s)                                                                                       | Change Summary                                                                                                         |
|----------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
| components/clp-py-utils/clp_py_utils/clp_config.py                                           | Added five string constants for table suffixes: `ARCHIVE_TAGS_TABLE_SUFFIX`, `COLUMN_METADATA_TABLE_SUFFIX`, `DATASETS_TABLE_SUFFIX`, `FILES_TABLE_SUFFIX`, and `TAGS_TABLE_SUFFIX`. |
| components/clp-py-utils/clp_py_utils/clp_metadata_db_utils.py                                | New module introducing functions for creating metadata tables, including helpers for individual tables and a coordinator function.                |
| components/clp-py-utils/clp_py_utils/initialize-clp-metadata-db.py                           | Refactored to use `create_metadata_db_tables` function replacing multiple explicit SQL `CREATE TABLE` statements; added required `--storage-engine` argument; converts config path to `pathlib.Path`; calls `create_datasets_table` if storage engine is `CLP_S`. |
| components/clp-py-utils/clp_py_utils/create-db-tables.py                                    | Added required `--storage-engine` command-line argument validated against `StorageEngine` enum; passes this argument to subprocess call of initialization script. |
| components/clp-package-utils/clp_package_utils/scripts/start_clp.py                          | Updated invocation of `create-db-tables.py` script to include the `--storage-engine` argument with configured storage engine value. |

## Sequence Diagram(s)

```mermaid
sequenceDiagram
    participant StartScript as start_clp.py
    participant CreateScript as create-db-tables.py
    participant InitScript as initialize-clp-metadata-db.py
    participant Utils as clp_metadata_db_utils.py
    participant DB as Database

    StartScript->>CreateScript: Run with --storage-engine argument
    CreateScript->>InitScript: Run with --storage-engine argument
    InitScript->>Utils: create_metadata_db_tables(metadata_db_cursor, table_prefix, dataset?)
    Utils->>DB: CREATE TABLE IF NOT EXISTS datasets
    Utils->>DB: CREATE TABLE IF NOT EXISTS archives
    Utils->>DB: CREATE TABLE IF NOT EXISTS tags
    Utils->>DB: CREATE TABLE IF NOT EXISTS archive_tags
    Utils->>DB: CREATE TABLE IF NOT EXISTS files
    alt If dataset specified (e.g., for CLP_S storage engine)
        Utils->>DB: CREATE TABLE IF NOT EXISTS column_metadata
    end
    InitScript->>DB: Commit transaction
```

<!-- walkthrough_end -->
<!-- internal state start -->


<!-- DwQgtGAEAqAWCWBnSTIEMB26CuAXA9mAOYCmGJATmriQCaQDG+Ats2bgFyQAOFk+AIwBWJBrngA3EsgEBPRvlqU0AgfFwA6NPEgQAfACgjoCEYDEZyAAUASpETZWaCrKNxU3bABsvkCiQBHbGlcABpIcVwvOkgAIn8AMzQxfAoACgYvbjBuWTA8eC9EAEouAFl8KUgAYQAZK0g2XDRaajRIABEAIQiVaLAGf2p4fCwvfCJ4BhQMAkgCr3V5ZkVvEgBuGtHadRGMNB95QZJqEgjYM9bmxBJcZGaBaMgAdwusbER4DCJIAANM7KIX6hWIzXAUVYMaT2EhSKi+BiwTCkZD+LynehzK4qNA3XqPM7HYajSAJVLnM51Bq8fAiMQaIwAaRIRyR32hX0y2CUHCMUAAgrQduISfgEuhSdgMGI9hF8IwhjQKX8AQB9bE3O6/fHRBlQGwkJIpChfH64C4Kk4isYTKak8kSZwjD41eqNW4tNo6jmzeXtfwfPpnKyyc0khZLRqrXUGKAASVmENo2BlovF7QuWUokul1rlltOEs1/HFD2iyGe6lg6Cw+G41oOfw1t213GcaCa2fN1EgSiS3ju+d+ADlRiRfrzILpIHH0022prtahENxRPAEvA6OFzWcyT58JXvt7kM5CYqYpXzcrm7hIPs2OgTzxEvAAB6TqfT/5ZVUAbxvAC+qrOIikjSL8Bifl+ar/gutxAc0RBApBn5QN+3B/oBwEUKBUiqohyFQWhMFYRu5YQURKo/rB1zwaqTBeI4GCqk0nrNBR06zne8q/DeS7ICua4blu16GmgA7Hug/gFjQtAflOxE/iBCBSIRqFURhBEcQpGnYbhJD4WgSHadBP5keBsYwBa6H0fgjHMMxrHYuqYkDtqZZnKgGDyuM7J8ESsmQHIvauV4miWXAZy/BAiAEFQpBgGQkzkNqCToj8Hi4jc9AJBCzB/LFzi4AMWQaLk7k8V8uwHPAABeJAldkTltGAtACGVsjAjWmLWWqN5At6KDIAFMSjF48jrsqsWpEZZxJV8nnIOhYBAgyBjuMgtj2CmUJ0MgZgABwAIwAJzdSgzDcNEbCzMgbYUIOYrXnBt4JFa2DSVdmDhIJDDrlMBzjbyEUWgFspPTuun9e5QaQGkADKNDcJAABMxR6pAABiySFOowxHojJDIwAzEF8hkH0poyeD4rkM8knYNwJJkEwUo0CaR7tHTvYvWtABqNVXLKzy4jwlBkhQbD0B8VOQ8tbYMAA1rNXVMBgG6S7LSK3h5J7SSN9AOAwUKIIgCTeONj6QCQr6rmIMQS0F+BXpDfbiWFroNJgPVRQCK3atN8Vzd8C32FC+wmvgiAaFZhKjOC8ACAUJJMJdizQpe1aQzSdK4AA5MN8cmkneZENg8BKIs5Anhg9C0Pg0LebeXzgpCZwCEMitU4iyLSDHHT4AwjjsMS7zcFcGeUHNVbZswLRnOSSgkFL8wYOQJuIM4shrRYWysOwAmOHPLhuBaY1HCwN23iS5oeBbfiBMEsUvKLc9KGTyofNmvxdxQitJiaMu4EY6RXkFQJAMR2hqyhBQLA/giDOB2EeSGNsmYOGkhDayEA+L8HrLKOYkMv4UBjvDO2/0GCA1kNuC48ggghD2BeWA8g5bYJetqOseYRYyBIFTN+Zw0COkKLDHst8BKIRIOEJuWsezJg5j8dovBYTOmQGwW4VN1AvxPEQfwMRuw6wQMgDhspECwHwN4egTcgpzVtlHGIbNCjKlTldW4lwXqknetJVA5tDgPxaNvIwo5JQPQuP5S+B9wg7EQEPU2exEDhHJP4FcowbgVing/JgFAlD0BvgYng99/B0NiiAi04kwx8HJM7EJChEyJzwKkZAOwLHO2fJUCuhJ0Qmg3BQ60hi+AtFoIk+4uToGUCwJnL4yoBHaHRASBQ+9bprX0MYcAUAyDZPTHU4gZBlCBVTlfLgvB+DCFEOIVSH8mBKCoKodQWgdBLJMFADaQ0aw4AIFs8gVBdlhNmFwKg9MHBOBcOcxQyhrmaG0LoMAhhlmmAME4sct0AD0ftcj5HEEUZFP5ciqgWIgTFGE1YbiIB1XksQyUGF3vyOM7ydkxABcfeQT0e7skQEYTGYE7wkH+QnI8atCq3Qfoom4sxZawwcAkDcr5J7SX6XQLgvx+Q2GqAACTjHzAAoqqaA/IADi8MtX8i6LUTV8MACqmNMZxgABrakzpAR0jEoqxGUmBQySFYhdV+NUAA8rUU1ZRhyqjKOq7VHR+TaoNUak15rLU2peFWe1Bxgh/FiAxJiLEPTYg9eEX4YbtXwxDfq7VUbVRmotda21CaHXJt+LEfq2a/iWuNUWw1xrS0xorfGq81anXmUQA272fxtV6sjW2stsbK3dqTU6giHrinSDjhgflg5nhpJbkmFMkDfJITacqG2SBxBHgVUq1VGqW0lvHZ2vlzRZicucMqAgyNMG7kKCQGOgTKnZguRIyA4xJgMHiXwc2uY9g1VwPILhUYdjCVoGtcwlh+RhR2bE/MrtRAdNHoY8UqDUiBXJJ4R4dp2C7GkHySAgosnVOXX8RVKq1WauHeesdHarVcFinwAAvHEF1eFZ1LiwP8FgTNyBIpRXkXF+LVTYok2qQl8BiXlUshRmI17MC3i9b6/1gbg2hvDfyUd0by2sfsOCSAXHU12XTc1ZoHqZgqiEwiu4+KcjifRXitU0m3OSbkwpzqSmhQqeSTe9Teb+QFugExwzsa2OmfM/W/j9nLqOfc1kFzaLCgpYwp5jL3nRhEo6hBAUAX6Cqdvb8Jt6rIvtqMzFzjcQ+22YmYJpLImnNifSxijzsgcVedk3l+TBX/OUdK+pxjBnqvRZM3V2IfG7PNeEwfZzqKZNYu6ytgl/XfMQTJbEPksL4Wtcy2l9bUm1teaquIGq9VGpgGs2gVq7VcikvJZS6lpAPkYm2oCxl4pmUolPmcKJJp6zcXpgU8uiSJTcznrXagqQL5OFrmAKu/CKBlyvn8GKcVZqJRDilfMv11zMItIHWa1s8c/odRXT7RltBLv0VFRGM1SDqop9qMgjh508weLiRdRLSSvp4NQasqA1ZwkCnMdovw2zmkWO1Kwwv2HHLEFYiWZxxgtFNFzxREhlHW1tosP6t54YAEVah/GqDYdV4b1UwFbbbuMmNIDDm9dASA6qrVxnhhFgOzQaBX2QI7ZgA54DOPdM0bEkkkRVAECQMgD8VhSHoIO/wX0oT0GChQnw9x5SQ3gElh6MR/jngzRHto6oBCGQJANYDqYMAxwo7sUYlDwiTUhjcaI9tDbY9IOT5Ki0Lf1FLV1dvgxQ+3iz0UFUJf+pV+iNqWPavqZHju8eBvMBvUdG9XMjHZiogLUD3gD6gOkaohIOiCX8obzuOoMfi6zir6j3tP5VLQhECjC53sjRg7KAQj4GyWgKuH4P9O0fwOeCZKUP7OgODClBDJDT5FDAhC0JQTIdsHpEsfXJmQvbJPgAjQ3cnSITcVlAwUccgBkHbPbMAIwA7RbdrE7bLTrH8O7CvHrDLElAwCg17GlT5OlI+LeDAqA4g/kTlemFYZMJ4GyZgtqVgooArMEDdE2HMOvHPGSfhGEa+cUNET7VfXWOzdoMoJ0NAbofgPgMoWQU3c3bEAQXnGOOMW8PsA/GYHYXXZMRsTMVcIDSA9AuYEaSSR2HjaQbcIyOJKSfSMAAiB+C/WJBAbgEIvtcINNBycPNiNAcIQdfqGOdVZIasWvPMG2UQPAaEKXS3a3aAW3YtY1GcJ3F3N3D3L3H3SACwkzU4DHO1QnLpMOC4OeEKDcDAJvJdVvaURiJBH4XgfPfgxWFkEIqUeAOhSASY2QOI1IHhIgLABYkIwdL4JQKVaOWOafK0AyWfDyVKLw2UEaJ8a/L4CWOePMDyLtHIzcLwWgQPReNxe8fhWuUIlSE/ZnS4eAfwY0bePY4vA40vFIlg3WE4kDUUHCC4djU4IZM8K0GmCkPEXWeJPBZvHxRRSVUVWZd4isBNdoS4pQEVYSPpT4sGZBUGSzJInQ2GO1Yk0KewtxcVSVLnO4qJLo0WJgOEAXc/Z47aREK2OMDoEI8QNgQqS6EIz4eqEImXWADYz4rQnpGI7hXAVdePAImuTEYIz/EFbabgLAwcAQSpSAAwk0IwnoQdMwpooeCgd/PgcDVcF4vgfIoeQ9H4O0y+b2aOeDcjeAzDVDZA9DNAlDJ6HDbAkwvJQjaYYjcQUjSyTGU4kkWVOSfY04dUF6RAOfEgNIaQ+0x0oIgkKTF8d8KbYoXQPQZ3McBLGyBgzLME5yaQ3FQbKAZM6ErANM+VEaJs8vaQ3WfMyvQs1IYs6IUsw0N8WrCJF6WrSAAAHxrPIDMyXJIErKhVXLrK6xkMbKkMr1bMUwoNjH2wc0OyW2SGVgSnoIvNmh3MRSB3HzxUKgenolKiew4JewQze22R4MNj4KBSZTZH+3WlBiAv4SFBrnQCFH6MbD2W9mR1DmcHR3YExxWh7wanmnxztUhh7QwME1mDp0oFfI23VgGwVkvLfVJ1IFVEwvHGDM8gwAkEHif2fQzJoAe3CKDGjnKjDmB2biXV3Tll7IHK4qhLr2KVQDEP+mhEhjgs+L0T8ClERO5xxDRNhjBhTnjkIr4GClQUN3UEtjbBiWpJ+KDj71Dh8w+ifyQMBzH3rHfXlE/T4BAIA2tgoD/0gAAKAMA2qVbl8DSgPGeTQAlROWgKMH9MQ3ZiDJspClQIQOSQwMjLw1wOwFjIIJI2INIJIA/N22PKoLhVPNoNS2W161WzvJGg4t1nYM4K/O4M+3pX4MAt7mIMil4vH1B3v1w2Ut+CZyDlZ373ZwwEcFJDyl0gbI0D61IqIF+H1MR1oAQuXKQuHlvXuhuD4FQBthoFrl0XlHB3+LUO5mijQt+NxwGqkmQtmHCFeCmGrGD2flj34GXNYvYyph7QaUnPIHoAmTll6tmn6oWkGs5z2NWrpXQvMuXJeqPFF1GHFxiGsKVjBD9D+F+pZzZ3J2Gq+H5ShAkuQA53ylwtQB3CwCMuyitkwCguFFA18CWoxyOqoowrRpiocAEBpA3kYEBnOB7AoCUr+Au3gCuwaj9juwewKwcv4B3D4GZtZukGGkBl6V/VtGmFPEYDAtg3CtgIDKiu8NzxDLiqDIjJsSjPw1SvwPjKILI2U2TxEIfiCH2pKx9KRxR3OuWvUyxxOtovYVLFkFXAKnBEnWrESQTi7xVvwCmGhFyhYGRvQv+vxzxvor+HAIwDSCQokGKDErzCaxoNE2Ktcxy23Jk3PEqq4tFpyr22hQeXJ3WVeUIHe1pXtvmU4D8DQH+X/PkEzxBSuTUHBTuShQMArq/1wFVArhzP8F1y5ToFVGfNvCWT7pWUgGJgSFOhRmJgABYBBaBiY6AToBAAB2FGWgBIA6AABlaBOjQBXqOgYBRh3uSFoBXrQBIFjxIDPshUMAroAFYJUGAEgToUZH6DpRBz616d6SAjoN6UYToj60Ad7aAAA2Fe2BgQI+gQE6Y2A6ZIFe1+2eiASAE6WgPBk+2Bg+nenek6R+4mI+te2gBgA6Feo+zeo+hgd+4mc+gBsBihA6WBnerBiuo+o6WBo6A6Q+o+kho+/elGWBk6A6ChtAFho6AQFGWPE6WBxBzhk4OPI6RRnhuele9+lGd+g6KEd+uB2BtAA6Heih++g6Ze0h/Bk6FhnesRleiBzegBk6I6bRnBk6FeihlGI+pIB+o+oh0xtACRhIHeg6Wh6BhB2gER5+4mA6NqY+pBzxqAIh+R4md+0x5+lekgUhiB8xtqWRxJixhIYx6wk6VB2BzJ2gCJ1J62BgI6IJ6wleqRw0JB6Ro+9+i+9+9++eBgPhw0G+spg6bxsRs4e5OeloLhhIfBkgPho6TekhuhqEI+oJ8+hIXJo+yJ2gSJmh2Bz+x++pgQWBv+iRtAfxgRkge+hILhlR4mBeqBgBnegZhgXepZ8JhgRB+pgxu+oh8+4mI6cJzehIfRgQI6BBihVQBgTJhRhgFe6hgQYmX+ne2B+p4mFGGFyRxF9+hRzF0F2JoB+BlGOhgQQxu+3F0B2hwR9oSZnB2huPCxne7p4mHei+tezh1oA5+hhgMhxp4l9+tlje3+lGA6ep2B2Z/R4mbFkgFGN6fJw+2psBtAVB2hpQeRlB+Bk5lGUJ+pk4C53p9Zle2Zn+2Bg6d+vJk4fxxJlhzhk+zJnVo6UQBB45phuPWBkgWh3Rte5+vRohkh8J9BloZerhoJrphRlQFGeps+6xwVkR9+xh/B9+3+5B9B955x6w51vJ0NxZiVs6Ge/uy+dQIe541UUezcVdWgGiz4/QIAA=== -->

<!-- internal state end -->
<!-- finishing_touch_checkbox_start -->

<details>
<summary>✨ Finishing Touches</summary>

- [ ] <!-- {"checkboxId": "7962f53c-55bc-4827-bfbf-6a18da830691"} --> 📝 Generate Docstrings

</details>

<!-- finishing_touch_checkbox_end -->
<!-- tips_start -->

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

<details>
<summary>❤️ Share</summary>

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai)
- [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai)
- [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai)
- [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

</details>

<details>
<summary>🪧 Tips</summary>

### Chat

There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai?utm_source=oss&utm_medium=github&utm_campaign=y-scope/clp&utm_content=831):

- Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  - `I pushed a fix in commit <commit_id>, please review it.`
  - `Generate unit testing code for this file.`
  - `Open a follow-up GitHub issue for this discussion.`
- Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples:
  - `@coderabbitai generate unit testing code for this file.`
  -	`@coderabbitai modularize this function.`
- PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  - `@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.`
  - `@coderabbitai read src/utils.ts and generate unit testing code.`
  - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.`
  - `@coderabbitai help me debug CodeRabbit configuration file.`

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

### CodeRabbit Commands (Invoked using PR comments)

- `@coderabbitai pause` to pause the reviews on a PR.
- `@coderabbitai resume` to resume the paused reviews.
- `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
- `@coderabbitai full review` to do a full review from scratch and review all the files again.
- `@coderabbitai summary` to regenerate the summary of the PR.
- `@coderabbitai generate docstrings` to [generate docstrings](https://docs.coderabbit.ai/finishing-touches/docstrings) for this PR.
- `@coderabbitai generate sequence diagram` to generate a sequence diagram of the changes in this PR.
- `@coderabbitai resolve` resolve all the CodeRabbit review comments.
- `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository.
- `@coderabbitai help` to get help.

### Other keywords and placeholders

- Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed.
- Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description.
- Add `@coderabbitai` anywhere in the PR title to generate the title automatically.

### CodeRabbit Configuration File (`.coderabbit.yaml`)

- You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository.
- Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information.
- If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json`

### Documentation and Community

- Visit our [Documentation](https://docs.coderabbit.ai) for detailed information on how to use CodeRabbit.
- Join our [Discord Community](http://discord.gg/coderabbit) to get help, request features, and share feedback.
- Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

</details>

<!-- tips_end -->
author_association: CONTRIBUTOR
user: login: kirkrodrigues
created_at: 2025-04-17T18:03:39Z
body: Why are we exposing the `--dataset` option to the user? I thought we discussed during our meeting yesterday that we *don't* want to do that until the complete feature is ready, right?
author_association: MEMBER