Repository: Sage-Bionetworks-Workflows/orca-recipes
Pull request #89: [DPE-1288] ALS-KP DAG
Author: login: BWMac
Created: 2025-05-08T17:28:08Z | Merged: 2025-05-12T14:46:20Z

# **Problem:**

We need an automated solution for retrieving Dataset metadata from the FAIR C-Path data portal, transforming it to our core data model and loading it into a Dataset Collection in Synapse to power the new federated ALS data portal.

# **Solution:**

This PR implements an Airflow DAG that runs at midnight on the first of each month and performs the following steps:

1. Fetch data from the C-Path API using an authentication token stored in Airflow Variables
2. Transform the raw data using a JSONata mapping expression and validate against a JSON Schema
3. Create or update Synapse Datasets for each item in the transformed data
4. Create or update a Dataset Collection containing all the datasets
5. Create or update annotations for each dataset in the collection
6. Create a new snapshot of the collection if any changes were detected

**Note:** The specific details and data model are subject to change if requested by our stakeholders. This PR will not be merged until these details are confirmed.

# **Testing:**

The DAG was run manually to generate the Dataset Collection that can be viewed on Synapse [here](https://www.synapse.org/Synapse:syn66496326/datasets/).


Diff:
diff --git a/dags/als-kp-dataset-dag.py b/dags/als-kp-dataset-dag.py
new file mode 100644
index 00000000..464006de
--- /dev/null
+++ b/dags/als-kp-dataset-dag.py
@@ -0,0 +1,387 @@
+"""ALS Knowledge Portal Dataset Collection DAG
+
+This DAG automates the process of creating and maintaining a Synapse Dataset Collection
+for the ALS Knowledge Portal. The DAG follows these steps:
+
+1. Fetch data from the C-Path API using an authentication token stored in Airflow Variables
+2. Transform the raw data using a JSONata mapping expression and validate against a JSON Schema
+3. Create or update Synapse Datasets for each item in the transformed data
+4. Create or update a Dataset Collection containing all the datasets
+5. Create or update annotations for each dataset in the collection
+6. Create a new snapshot of the collection if any changes were detected
+
+The DAG runs monthly and uses Airflow Variables for configuration.
+"""
+
+from datetime import datetime
+import requests
+from typing import Dict, List, Tuple, Any, Optional
+
+import pandas as pd
+from jsonata import jsonata
+from jsonschema import validate, ValidationError
+
+from synapseclient.models import Dataset, DatasetCollection, Column, ColumnType
+from orca.services.synapse import SynapseHook
+
+from airflow.decorators import task, dag
+from airflow.models import Variable, Param
+
+
+dag_params = {
+    "project_id": Param("syn64892175", type="string"),
+    "mapping_url": Param(
+        "https://raw.githubusercontent.com/amp-als/data-model/dd0e476c3659c9b98977d567d0ddce01d5057639/mapping/cpath.jsonata",
+        type="string",
+    ),
+    "schema_url": Param(
+        "https://raw.githubusercontent.com/amp-als/data-model/dd0e476c3659c9b98977d567d0ddce01d5057639/json-schemas/Dataset.json",
+        type="string",
+    ),
+    "cpath_api_url": Param(
+        "https://fair.dap.c-path.org/api/collections/als-kp/datasets", type="string"
+    ),
+    "collection_name": Param("Dataset collection (Production)", type="string"),
+    "collection_description": Param(
+        "A collection of datasets curated for the ALS Knowledge Portal", type="string"
+    ),
+    "synapse_conn_id": Param("SYNAPSE_ORCA_SERVICE_ACCOUNT_CONN", type="string"),
+}
+
+dag_config = {
+    "schedule_interval": "0 0 1 * *",  # Run on the first day of the month at midnight
+    "start_date": datetime(2025, 5, 1),
+    "catchup": False,
+    "default_args": {
+        "retries": 2,
+    },
+    "tags": ["als-kp"],
+    "params": dag_params,
+}
+
+
+def load_mapping_from_url(url: str) -> str:
+    """Load the JSONata mapping expression from a URL.
+
+    Arguments:
+        url (str): The URL to fetch the JSONata mapping expression from.
+
+    Returns:
+        str: The JSONata mapping expression as a string.
+
+    Raises:
+        requests.exceptions.RequestException: If the request fails or returns a non-200 status code.
+        requests.exceptions.Timeout: If the request times out.
+    """
+    response = requests.get(url, timeout=30)
+    response.raise_for_status()
+    return response.text
+
+
+def load_schema_from_url(url: str) -> Dict[str, Any]:
+    """Load the JSON Schema from a URL.
+
+    Arguments:
+        url (str): The URL to fetch the JSON Schema from.
+
+    Returns:
+        Dict[str, Any]: The parsed JSON Schema as a dictionary.
+
+    Raises:
+        requests.exceptions.RequestException: If the request fails or returns a non-200 status code.
+        requests.exceptions.Timeout: If the request times out.
+        json.JSONDecodeError: If the response is not valid JSON.
+    """
+    response = requests.get(url, timeout=30)
+    response.raise_for_status()
+    return response.json()
+
+
+def validate_item(
+    item: Dict[str, Any], schema: Dict[str, Any]
+) -> Tuple[bool, Optional[str]]:
+    """Validate an item against a JSON Schema.
+
+    Arguments:
+        item (Dict[str, Any]): The item to validate.
+        schema (Dict[str, Any]): The JSON Schema to validate against.
+
+    Returns:
+        Tuple[bool, Optional[str]]: A tuple containing:
+            - bool: True if the item is valid, False otherwise.
+            - Optional[str]: Error message if validation fails, None if validation succeeds.
+    """
+    try:
+        validate(instance=item, schema=schema)
+        return True, None
+    except ValidationError as e:
+        return False, str(e)
+
+
+def transform_with_jsonata(
+    source_items: List[Dict[str, Any]],
+    mapping_expr: str,
+    schema: Optional[Dict[str, Any]] = None,
+) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
+    """Transform a list of items using a JSONata expression and validate against schema.
+
+    Arguments:
+        source_items (List[Dict[str, Any]]): List of source items to transform.
+        mapping_expr (str): The JSONata mapping expression to apply.
+        schema (Optional[Dict[str, Any]], optional): JSON Schema to validate transformed items against.
+            If None, no validation is performed. Defaults to None.
+
+    Returns:
+        Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]: A tuple containing:
+            - List[Dict[str, Any]]: List of successfully transformed and validated items.
+            - List[Dict[str, Any]]: List of validation errors, each containing:
+                - item_index (int): Index of the item that failed validation
+                - error (str): Validation error message
+                - transformed_item (Dict[str, Any]): The transformed item that failed validation
+
+    Raises:
+        jsonata.JsonataError: If the JSONata expression is invalid.
+    """
+    expr = jsonata.Jsonata(mapping_expr)
+    transformed_items: List[Dict[str, Any]] = []
+    validation_errors: List[Dict[str, Any]] = []
+
+    for i, item in enumerate(source_items):
+        result = expr.evaluate(item)
+        if schema:
+            is_valid, error = validate_item(result, schema)
+            if not is_valid:
+                validation_errors.append(
+                    {"item_index": i, "error": error, "transformed_item": result}
+                )
+                continue
+        transformed_items.append(result)
+
+    return transformed_items, validation_errors
+
+
+@dag(**dag_config)
+def als_kp_dataset_dag():
+    @task
+    def fetch_cpath_data(**context) -> Dict[str, Any]:
+        """Fetch data from C-Path API using auth token from Airflow Variables.
+
+        Arguments:
+            **context: Airflow task context containing DAG parameters
+
+        Returns:
+            Dict[str, Any]: JSON response from the C-Path API containing dataset items
+
+        Raises:
+            requests.exceptions.RequestException: If the API request fails
+        """
+        headers = {
+            "accept": "application/json",
+            "Authorization": f"Bearer {Variable.get('CPATH_API_TOKEN')}",
+        }
+
+        response = requests.get(context["params"]["cpath_api_url"], headers=headers)
+        response.raise_for_status()
+        return response.json()
+
+    @task
+    def transform_data(data: Dict[str, Any], **context) -> List[Dict[str, Any]]:
+        """Transform the data using JSONata mapping and validate against schema.
+
+        This task:
+        1. Loads the JSONata mapping expression from the specified URL
+        2. Loads the JSON Schema from the specified URL
+        3. Applies the mapping to each item in the input data
+        4. Validates each transformed item against the schema
+        5. Returns only the valid transformed items
+
+        Arguments:
+            data: Raw data from the C-Path API
+            **context: Airflow task context containing DAG parameters
+
+        Returns:
+            List[Dict[str, Any]]: List of transformed and validated items
+
+        Raises:
+            ValueError: If any validation errors are found
+            requests.exceptions.RequestException: If loading mapping or schema fails
+        """
+        mapping_expr = load_mapping_from_url(context["params"]["mapping_url"])
+        schema = load_schema_from_url(context["params"]["schema_url"])
+
+        transformed_items, validation_errors = transform_with_jsonata(
+            data["items"], mapping_expr, schema
+        )
+
+        if validation_errors:
+            raise ValueError(f"Found {len(validation_errors)} validation errors.")
+
+        return transformed_items
+
+    @task
+    def create_datasets(
+        transformed_items: List[Dict[str, Any]], **context
+    ) -> DatasetCollection:
+        """Create Synapse datasets and collection.
+
+        This task:
+        1. Creates a new Dataset Collection with predefined columns
+        2. For each transformed item:
+           - Creates a new Dataset with the item's title and description
+           - Adds the Dataset to the collection
+        3. Stores the collection in Synapse
+
+        Arguments:
+            transformed_items: List of transformed and validated items
+            **context: Airflow task context containing DAG parameters
+
+        Returns:
+            DatasetCollection: The created and stored dataset collection
+        """
+        syn_hook = SynapseHook(context["params"]["synapse_conn_id"])
+        synapse_client = syn_hook.client
+
+        columns = [
+            Column(name="id", column_type=ColumnType.ENTITYID),
+            Column(name="title", column_type=ColumnType.STRING, maximum_size=200),
+            Column(name="creator", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="keywords", column_type=ColumnType.STRING, maximum_size=250),
+            Column(name="subject", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="collection", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="publisher", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="species", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="sameAs", column_type=ColumnType.STRING, maximum_size=100),
+        ]
+
+        dataset_collection = DatasetCollection(
+            name=context["params"]["collection_name"],
+            description=context["params"]["collection_description"],
+            parent_id=context["params"]["project_id"],
+            include_default_columns=True,
+            columns=columns,
+        )
+
+        for item in transformed_items:
+            dataset_description = (
+                item["description"][:1000]
+                if len(item["description"]) > 1000
+                else item["description"]
+            )
+            dataset = Dataset(
+                name=item["title"],
+                description=dataset_description,
+                parent_id=context["params"]["project_id"],
+            ).store(synapse_client=synapse_client)
+            dataset_collection.add_item(dataset)
+
+        dataset_collection = dataset_collection.store(synapse_client=synapse_client)
+        return dataset_collection
+
+    @task
+    def update_annotations(
+        dataset_collection: DatasetCollection,
+        transformed_items: List[Dict[str, Any]],
+        **context,
+    ) -> None:
+        """Update dataset annotations and create snapshot if changes are detected.
+
+        This task:
+        1. Queries the current state of the dataset collection
+        2. Prepares new annotation data from the transformed items
+        3. Updates the collection with the new annotations
+        4. Queries the updated state
+        5. Compares the before and after states
+        6. If changes are detected:
+           - Creates a summary of which columns were modified
+           - Creates a new snapshot with a descriptive comment
+           - The comment includes timestamp, changed columns, and dataset count
+
+        Arguments:
+            dataset_collection: The dataset collection to update
+            transformed_items: List of transformed items containing new annotations
+            **context: Airflow task context containing DAG parameters
+        """
+        syn_hook = SynapseHook(context["params"]["synapse_conn_id"])
+        synapse_client = syn_hook.client
+
+        dataset_ids = [item.id for item in dataset_collection.items]
+
+        # Get current data before update
+        current_data = dataset_collection.query(
+            query=f"SELECT * from {dataset_collection.id}",
+            synapse_client=synapse_client,
+        )
+        # Prepare and apply new data
+        annotation_data = pd.DataFrame(
+            {
+                "id": dataset_ids,
+                **{
+                    key: [
+                        (
+                            ", ".join(item[key])
+                            if isinstance(item[key], list)
+                            else item[key]
+                        )
+                        for item in transformed_items
+                    ]
+                    for key in [
+                        "title",
+                        "creator",
+                        "keywords",
+                        "subject",
+                        "collection",
+                        "publisher",
+                        "species",
+                        "sameAs",
+                    ]
+                },
+            }
+        )
+
+        # Update the rows
+        dataset_collection.update_rows(
+            values=annotation_data,
+            primary_keys=["id"],
+            dry_run=False,
+            wait_for_eventually_consistent_view=True,
+            synapse_client=synapse_client,
+        )
+
+        # Get data after update
+        updated_data = dataset_collection.query(
+            query=f"SELECT * from {dataset_collection.id}",
+            synapse_client=synapse_client,
+        )
+
+        # Compare data before and after update
+        if not current_data.equals(updated_data):
+            # Generate change summary
+            changed_columns = []
+            for col in current_data.columns:
+                if not current_data[col].equals(updated_data[col]):
+                    changed_columns.append(col)
+
+            # Create snapshot comment with timestamp and change summary
+            timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
+            snapshot_comment = (
+                f"Snapshot created at {timestamp}. "
+                f"Updated columns: {', '.join(changed_columns)}. "
+                f"Total datasets: {len(dataset_ids)}"
+            )
+
+            print("Changes detected in dataset collection, creating new snapshot...")
+            print(f"Snapshot comment: {snapshot_comment}")
+            dataset_collection.snapshot(
+                comment=snapshot_comment, synapse_client=synapse_client
+            )
+        else:
+            print("No changes detected in dataset collection, skipping snapshot.")
+
+    # Define task dependencies
+    data = fetch_cpath_data()
+    transformed_items = transform_data(data)
+    dataset_collection = create_datasets(transformed_items)
+    update_annotations(dataset_collection, transformed_items)
+
+
+als_kp_dataset_dag()
diff --git a/requirements-airflow.txt b/requirements-airflow.txt
index 92b2091d..b0702241 100644
--- a/requirements-airflow.txt
+++ b/requirements-airflow.txt
@@ -2,7 +2,7 @@ apache-airflow-providers-amazon ~=9.2.0
 apache-airflow-providers-celery >=3.10.0
 apache-airflow-providers-snowflake ~=6.0.0
 snowflake-connector-python ~=3.13.2
-py-orca[all] == 1.5.1
+py-orca[all] == 1.5.2
 pandas <3.0.0
 slack-sdk >=3.27
 pendulum~=3.0.0
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 2ad75b41..13caa315 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -1,5 +1,6 @@
-synapseclient >=4.7,<5.0
+synapseclient >=4.8,<5.0
 fs-synapse >=2.0,<3.0
 s3fs ~=2023.5
 metaflow ~=2.9
 boto3 >=1.7.0,<2.0
+jsonata-python ~=0.5.3


Reviews:
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T18:53:16Z
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T18:53:32Z
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T18:53:59Z
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T18:57:02Z
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T18:59:08Z
user: login: BWMac
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T19:01:21Z
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T19:02:27Z
user: login: BryanFauble
body: Great work. I flagged a few things, but overall it looks good to me
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T19:04:02Z
user: login: thomasyu888
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T19:11:03Z
user: login: BWMac
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T19:13:17Z
user: login: thomasyu888
body: 🔥 Awesome work here!
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T06:02:33Z

Review comments:
diff_hunk: @@ -0,0 +1,338 @@
+"""ALS Knowledge Portal Dataset Collection DAG
+
+This DAG automates the process of creating and maintaining a Synapse Dataset Collection
+for the ALS Knowledge Portal. The DAG follows these steps:
+
+1. Fetch data from the C-Path API using an authentication token stored in Airflow Variables
+2. Transform the raw data using a JSONata mapping expression and validate against a JSON Schema
+3. Create or update Synapse Datasets for each item in the transformed data
+4. Create or update a Dataset Collection containing all the datasets
+5. Create or update annotations for each dataset in the collection
+6. Create a new snapshot of the collection if any changes were detected
+
+The DAG runs monthly and uses Airflow Variables for configuration.
+"""
+
+from datetime import datetime
+import requests
+from typing import Dict, List, Tuple, Any, Optional
+
+import pandas as pd
+from jsonata import jsonata
+from jsonschema import validate, ValidationError
+
+from synapseclient.models import Dataset, DatasetCollection, Column, ColumnType
+from orca.services.synapse import SynapseHook
+
+from airflow.decorators import task, dag
+from airflow.models import Variable, Param
+
+
+dag_params = {
+    "project_id": Param("syn64892175", type="string"),
+    "mapping_url": Param(
+        "https://raw.githubusercontent.com/amp-als/data-model/refs/heads/main/mapping/cpath.jsonata",
+        type="string",
+    ),
+    "schema_url": Param(
+        "https://raw.githubusercontent.com/amp-als/data-model/refs/heads/main/json-schemas/Dataset.json",
+        type="string",
+    ),
+    "cpath_api_url": Param(
+        "https://fair.dap.c-path.org/api/collections/als-kp/datasets", type="string"
+    ),
+    "collection_name": Param("Dataset collection (Production)", type="string"),
+    "collection_description": Param(
+        "A collection of datasets curated for the ALS Knowledge Portal", type="string"
+    ),
+    "synapse_conn_id": Param("SYNAPSE_ORCA_SERVICE_ACCOUNT_CONN", type="string"),
+}
+
+dag_config = {
+    "schedule_interval": "0 0 1 * *",  # Run on the first day of the month at midnight
+    "start_date": datetime(2025, 5, 1),
+    "catchup": False,
+    "default_args": {
+        "retries": 2,
+    },
+    "tags": ["als-kp"],
+    "params": dag_params,
+}
+
+
+def load_mapping_from_url(url: str) -> str:
+    """Load the JSONata mapping expression from a URL"""
+    response = requests.get(url)
path: dags/als-kp-dataset-dag.py
user: login: BryanFauble
body: Make sure a timeout is set when executing this call 
created_at: 2025-05-08T18:53:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,338 @@
+"""ALS Knowledge Portal Dataset Collection DAG
+
+This DAG automates the process of creating and maintaining a Synapse Dataset Collection
+for the ALS Knowledge Portal. The DAG follows these steps:
+
+1. Fetch data from the C-Path API using an authentication token stored in Airflow Variables
+2. Transform the raw data using a JSONata mapping expression and validate against a JSON Schema
+3. Create or update Synapse Datasets for each item in the transformed data
+4. Create or update a Dataset Collection containing all the datasets
+5. Create or update annotations for each dataset in the collection
+6. Create a new snapshot of the collection if any changes were detected
+
+The DAG runs monthly and uses Airflow Variables for configuration.
+"""
+
+from datetime import datetime
+import requests
+from typing import Dict, List, Tuple, Any, Optional
+
+import pandas as pd
+from jsonata import jsonata
+from jsonschema import validate, ValidationError
+
+from synapseclient.models import Dataset, DatasetCollection, Column, ColumnType
+from orca.services.synapse import SynapseHook
+
+from airflow.decorators import task, dag
+from airflow.models import Variable, Param
+
+
+dag_params = {
+    "project_id": Param("syn64892175", type="string"),
+    "mapping_url": Param(
+        "https://raw.githubusercontent.com/amp-als/data-model/refs/heads/main/mapping/cpath.jsonata",
+        type="string",
+    ),
+    "schema_url": Param(
+        "https://raw.githubusercontent.com/amp-als/data-model/refs/heads/main/json-schemas/Dataset.json",
+        type="string",
+    ),
+    "cpath_api_url": Param(
+        "https://fair.dap.c-path.org/api/collections/als-kp/datasets", type="string"
+    ),
+    "collection_name": Param("Dataset collection (Production)", type="string"),
+    "collection_description": Param(
+        "A collection of datasets curated for the ALS Knowledge Portal", type="string"
+    ),
+    "synapse_conn_id": Param("SYNAPSE_ORCA_SERVICE_ACCOUNT_CONN", type="string"),
+}
+
+dag_config = {
+    "schedule_interval": "0 0 1 * *",  # Run on the first day of the month at midnight
+    "start_date": datetime(2025, 5, 1),
+    "catchup": False,
+    "default_args": {
+        "retries": 2,
+    },
+    "tags": ["als-kp"],
+    "params": dag_params,
+}
+
+
+def load_mapping_from_url(url: str) -> str:
+    """Load the JSONata mapping expression from a URL"""
+    response = requests.get(url)
+    response.raise_for_status()
+    return response.text
+
+
+def load_schema_from_url(url: str) -> Dict[str, Any]:
+    """Load the JSON Schema from a URL"""
+    response = requests.get(url)
path: dags/als-kp-dataset-dag.py
user: login: BryanFauble
body: And this call 
created_at: 2025-05-08T18:53:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,338 @@
+"""ALS Knowledge Portal Dataset Collection DAG
+
+This DAG automates the process of creating and maintaining a Synapse Dataset Collection
+for the ALS Knowledge Portal. The DAG follows these steps:
+
+1. Fetch data from the C-Path API using an authentication token stored in Airflow Variables
+2. Transform the raw data using a JSONata mapping expression and validate against a JSON Schema
+3. Create or update Synapse Datasets for each item in the transformed data
+4. Create or update a Dataset Collection containing all the datasets
+5. Create or update annotations for each dataset in the collection
+6. Create a new snapshot of the collection if any changes were detected
+
+The DAG runs monthly and uses Airflow Variables for configuration.
+"""
+
+from datetime import datetime
+import requests
+from typing import Dict, List, Tuple, Any, Optional
+
+import pandas as pd
+from jsonata import jsonata
+from jsonschema import validate, ValidationError
+
+from synapseclient.models import Dataset, DatasetCollection, Column, ColumnType
+from orca.services.synapse import SynapseHook
+
+from airflow.decorators import task, dag
+from airflow.models import Variable, Param
+
+
+dag_params = {
+    "project_id": Param("syn64892175", type="string"),
+    "mapping_url": Param(
+        "https://raw.githubusercontent.com/amp-als/data-model/refs/heads/main/mapping/cpath.jsonata",
+        type="string",
+    ),
+    "schema_url": Param(
+        "https://raw.githubusercontent.com/amp-als/data-model/refs/heads/main/json-schemas/Dataset.json",
+        type="string",
+    ),
+    "cpath_api_url": Param(
+        "https://fair.dap.c-path.org/api/collections/als-kp/datasets", type="string"
+    ),
+    "collection_name": Param("Dataset collection (Production)", type="string"),
+    "collection_description": Param(
+        "A collection of datasets curated for the ALS Knowledge Portal", type="string"
+    ),
+    "synapse_conn_id": Param("SYNAPSE_ORCA_SERVICE_ACCOUNT_CONN", type="string"),
+}
+
+dag_config = {
+    "schedule_interval": "0 0 1 * *",  # Run on the first day of the month at midnight
+    "start_date": datetime(2025, 5, 1),
+    "catchup": False,
+    "default_args": {
+        "retries": 2,
+    },
+    "tags": ["als-kp"],
+    "params": dag_params,
+}
+
+
+def load_mapping_from_url(url: str) -> str:
+    """Load the JSONata mapping expression from a URL"""
path: dags/als-kp-dataset-dag.py
user: login: BryanFauble
body: Nit: Arguments and return in the docstring
created_at: 2025-05-08T18:53:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,338 @@
+"""ALS Knowledge Portal Dataset Collection DAG
+
+This DAG automates the process of creating and maintaining a Synapse Dataset Collection
+for the ALS Knowledge Portal. The DAG follows these steps:
+
+1. Fetch data from the C-Path API using an authentication token stored in Airflow Variables
+2. Transform the raw data using a JSONata mapping expression and validate against a JSON Schema
+3. Create or update Synapse Datasets for each item in the transformed data
+4. Create or update a Dataset Collection containing all the datasets
+5. Create or update annotations for each dataset in the collection
+6. Create a new snapshot of the collection if any changes were detected
+
+The DAG runs monthly and uses Airflow Variables for configuration.
+"""
+
+from datetime import datetime
+import requests
+from typing import Dict, List, Tuple, Any, Optional
+
+import pandas as pd
+from jsonata import jsonata
+from jsonschema import validate, ValidationError
+
+from synapseclient.models import Dataset, DatasetCollection, Column, ColumnType
+from orca.services.synapse import SynapseHook
+
+from airflow.decorators import task, dag
+from airflow.models import Variable, Param
+
+
+dag_params = {
+    "project_id": Param("syn64892175", type="string"),
+    "mapping_url": Param(
+        "https://raw.githubusercontent.com/amp-als/data-model/refs/heads/main/mapping/cpath.jsonata",
+        type="string",
+    ),
+    "schema_url": Param(
+        "https://raw.githubusercontent.com/amp-als/data-model/refs/heads/main/json-schemas/Dataset.json",
+        type="string",
+    ),
+    "cpath_api_url": Param(
+        "https://fair.dap.c-path.org/api/collections/als-kp/datasets", type="string"
+    ),
+    "collection_name": Param("Dataset collection (Production)", type="string"),
+    "collection_description": Param(
+        "A collection of datasets curated for the ALS Knowledge Portal", type="string"
+    ),
+    "synapse_conn_id": Param("SYNAPSE_ORCA_SERVICE_ACCOUNT_CONN", type="string"),
+}
+
+dag_config = {
+    "schedule_interval": "0 0 1 * *",  # Run on the first day of the month at midnight
+    "start_date": datetime(2025, 5, 1),
+    "catchup": False,
+    "default_args": {
+        "retries": 2,
+    },
+    "tags": ["als-kp"],
+    "params": dag_params,
+}
+
+
+def load_mapping_from_url(url: str) -> str:
+    """Load the JSONata mapping expression from a URL"""
+    response = requests.get(url)
+    response.raise_for_status()
+    return response.text
+
+
+def load_schema_from_url(url: str) -> Dict[str, Any]:
+    """Load the JSON Schema from a URL"""
+    response = requests.get(url)
+    response.raise_for_status()
+    return response.json()
+
+
+def validate_item(
+    item: Dict[str, Any], schema: Dict[str, Any]
+) -> Tuple[bool, Optional[str]]:
+    """Validate an item against a JSON Schema"""
+    try:
+        validate(instance=item, schema=schema)
+        return True, None
+    except ValidationError as e:
+        return False, str(e)
+
+
+def transform_with_jsonata(
+    source_items: List[Dict[str, Any]],
+    mapping_expr: str,
+    schema: Optional[Dict[str, Any]] = None,
+) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
+    """Transform a list of items using a JSONata expression and validate against schema"""
+    expr = jsonata.Jsonata(mapping_expr)
+    transformed_items: List[Dict[str, Any]] = []
+    validation_errors: List[Dict[str, Any]] = []
+
+    for i, item in enumerate(source_items):
+        result = expr.evaluate(item)
+        if schema:
+            is_valid, error = validate_item(result, schema)
+            if not is_valid:
+                validation_errors.append(
+                    {"item_index": i, "error": error, "transformed_item": result}
+                )
+                continue
+        transformed_items.append(result)
+
+    return transformed_items, validation_errors
+
+
+@dag(**dag_config)
+def als_kp_dataset_dag():
+    @task
+    def fetch_cpath_data(**context) -> Dict[str, Any]:
+        """Fetch data from C-Path API using auth token from Airflow Variables.
+
+        Args:
+            **context: Airflow task context containing DAG parameters
+
+        Returns:
+            Dict[str, Any]: JSON response from the C-Path API containing dataset items
+
+        Raises:
+            requests.exceptions.RequestException: If the API request fails
+        """
+        headers = {
+            "accept": "application/json",
+            "Authorization": f"Bearer {Variable.get('CPATH_API_TOKEN')}",
+        }
+
+        response = requests.get(context["params"]["cpath_api_url"], headers=headers)
+        response.raise_for_status()
+        return response.json()
+
+    @task
+    def transform_data(data: Dict[str, Any], **context) -> List[Dict[str, Any]]:
+        """Transform the data using JSONata mapping and validate against schema.
+
+        This task:
+        1. Loads the JSONata mapping expression from the specified URL
+        2. Loads the JSON Schema from the specified URL
+        3. Applies the mapping to each item in the input data
+        4. Validates each transformed item against the schema
+        5. Returns only the valid transformed items
+
+        Args:
+            data: Raw data from the C-Path API
+            **context: Airflow task context containing DAG parameters
+
+        Returns:
+            List[Dict[str, Any]]: List of transformed and validated items
+
+        Raises:
+            ValueError: If any validation errors are found
+            requests.exceptions.RequestException: If loading mapping or schema fails
+        """
+        mapping_expr = load_mapping_from_url(context["params"]["mapping_url"])
+        schema = load_schema_from_url(context["params"]["schema_url"])
+
+        transformed_items, validation_errors = transform_with_jsonata(
+            data["items"], mapping_expr, schema
+        )
+
+        if validation_errors:
+            raise ValueError(f"Found {len(validation_errors)} validation errors.")
+
+        return transformed_items
+
+    @task
+    def create_datasets(
+        transformed_items: List[Dict[str, Any]], **context
+    ) -> DatasetCollection:
+        """Create Synapse datasets and collection.
+
+        This task:
+        1. Creates a new Dataset Collection with predefined columns
+        2. For each transformed item:
+           - Creates a new Dataset with the item's title and description
+           - Adds the Dataset to the collection
+        3. Stores the collection in Synapse
+
+        Args:
+            transformed_items: List of transformed and validated items
+            **context: Airflow task context containing DAG parameters
+
+        Returns:
+            DatasetCollection: The created and stored dataset collection
+        """
+        syn_hook = SynapseHook(context["params"]["synapse_conn_id"])
+        synapse_client = syn_hook.client
+
+        columns = [
+            Column(name="id", column_type=ColumnType.ENTITYID),
+            Column(name="title", column_type=ColumnType.STRING, maximum_size=200),
+            Column(name="creator", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="keywords", column_type=ColumnType.STRING, maximum_size=250),
+            Column(name="subject", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="collection", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="publisher", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="species", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="sameAs", column_type=ColumnType.STRING, maximum_size=100),
+        ]
+
+        dataset_collection = DatasetCollection(
+            name=context["params"]["collection_name"],
+            description=context["params"]["collection_description"],
+            parent_id=context["params"]["project_id"],
+            include_default_columns=True,
+            columns=columns,
+        )
+
+        for item in transformed_items:
+            dataset_description = (
+                item["description"][:1000]
+                if len(item["description"]) > 1000
+                else item["description"]
+            )
+            dataset = Dataset(
+                name=item["title"],
+                description=dataset_description,
+                parent_id=context["params"]["project_id"],
+            ).store(synapse_client=synapse_client)
+            dataset_collection.add_item(dataset)
+
+        dataset_collection = dataset_collection.store(synapse_client=synapse_client)
path: dags/als-kp-dataset-dag.py
user: login: BryanFauble
body: It's awesome to see our changes in SYNPY get used!
created_at: 2025-05-08T18:57:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,338 @@
+"""ALS Knowledge Portal Dataset Collection DAG
+
+This DAG automates the process of creating and maintaining a Synapse Dataset Collection
+for the ALS Knowledge Portal. The DAG follows these steps:
+
+1. Fetch data from the C-Path API using an authentication token stored in Airflow Variables
+2. Transform the raw data using a JSONata mapping expression and validate against a JSON Schema
+3. Create or update Synapse Datasets for each item in the transformed data
+4. Create or update a Dataset Collection containing all the datasets
+5. Create or update annotations for each dataset in the collection
+6. Create a new snapshot of the collection if any changes were detected
+
+The DAG runs monthly and uses Airflow Variables for configuration.
+"""
+
+from datetime import datetime
+import requests
+from typing import Dict, List, Tuple, Any, Optional
+
+import pandas as pd
+from jsonata import jsonata
+from jsonschema import validate, ValidationError
+
+from synapseclient.models import Dataset, DatasetCollection, Column, ColumnType
+from orca.services.synapse import SynapseHook
+
+from airflow.decorators import task, dag
+from airflow.models import Variable, Param
+
+
+dag_params = {
+    "project_id": Param("syn64892175", type="string"),
+    "mapping_url": Param(
+        "https://raw.githubusercontent.com/amp-als/data-model/refs/heads/main/mapping/cpath.jsonata",
+        type="string",
+    ),
+    "schema_url": Param(
+        "https://raw.githubusercontent.com/amp-als/data-model/refs/heads/main/json-schemas/Dataset.json",
+        type="string",
+    ),
+    "cpath_api_url": Param(
+        "https://fair.dap.c-path.org/api/collections/als-kp/datasets", type="string"
+    ),
+    "collection_name": Param("Dataset collection (Production)", type="string"),
+    "collection_description": Param(
+        "A collection of datasets curated for the ALS Knowledge Portal", type="string"
+    ),
+    "synapse_conn_id": Param("SYNAPSE_ORCA_SERVICE_ACCOUNT_CONN", type="string"),
+}
+
+dag_config = {
+    "schedule_interval": "0 0 1 * *",  # Run on the first day of the month at midnight
+    "start_date": datetime(2025, 5, 1),
+    "catchup": False,
+    "default_args": {
+        "retries": 2,
+    },
+    "tags": ["als-kp"],
+    "params": dag_params,
+}
+
+
+def load_mapping_from_url(url: str) -> str:
+    """Load the JSONata mapping expression from a URL"""
+    response = requests.get(url)
+    response.raise_for_status()
+    return response.text
+
+
+def load_schema_from_url(url: str) -> Dict[str, Any]:
+    """Load the JSON Schema from a URL"""
+    response = requests.get(url)
+    response.raise_for_status()
+    return response.json()
+
+
+def validate_item(
+    item: Dict[str, Any], schema: Dict[str, Any]
+) -> Tuple[bool, Optional[str]]:
+    """Validate an item against a JSON Schema"""
+    try:
+        validate(instance=item, schema=schema)
+        return True, None
+    except ValidationError as e:
+        return False, str(e)
+
+
+def transform_with_jsonata(
+    source_items: List[Dict[str, Any]],
+    mapping_expr: str,
+    schema: Optional[Dict[str, Any]] = None,
+) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
+    """Transform a list of items using a JSONata expression and validate against schema"""
+    expr = jsonata.Jsonata(mapping_expr)
+    transformed_items: List[Dict[str, Any]] = []
+    validation_errors: List[Dict[str, Any]] = []
+
+    for i, item in enumerate(source_items):
+        result = expr.evaluate(item)
+        if schema:
+            is_valid, error = validate_item(result, schema)
+            if not is_valid:
+                validation_errors.append(
+                    {"item_index": i, "error": error, "transformed_item": result}
+                )
+                continue
+        transformed_items.append(result)
+
+    return transformed_items, validation_errors
+
+
+@dag(**dag_config)
+def als_kp_dataset_dag():
+    @task
+    def fetch_cpath_data(**context) -> Dict[str, Any]:
+        """Fetch data from C-Path API using auth token from Airflow Variables.
+
+        Args:
+            **context: Airflow task context containing DAG parameters
+
+        Returns:
+            Dict[str, Any]: JSON response from the C-Path API containing dataset items
+
+        Raises:
+            requests.exceptions.RequestException: If the API request fails
+        """
+        headers = {
+            "accept": "application/json",
+            "Authorization": f"Bearer {Variable.get('CPATH_API_TOKEN')}",
+        }
+
+        response = requests.get(context["params"]["cpath_api_url"], headers=headers)
+        response.raise_for_status()
+        return response.json()
+
+    @task
+    def transform_data(data: Dict[str, Any], **context) -> List[Dict[str, Any]]:
+        """Transform the data using JSONata mapping and validate against schema.
+
+        This task:
+        1. Loads the JSONata mapping expression from the specified URL
+        2. Loads the JSON Schema from the specified URL
+        3. Applies the mapping to each item in the input data
+        4. Validates each transformed item against the schema
+        5. Returns only the valid transformed items
+
+        Args:
+            data: Raw data from the C-Path API
+            **context: Airflow task context containing DAG parameters
+
+        Returns:
+            List[Dict[str, Any]]: List of transformed and validated items
+
+        Raises:
+            ValueError: If any validation errors are found
+            requests.exceptions.RequestException: If loading mapping or schema fails
+        """
+        mapping_expr = load_mapping_from_url(context["params"]["mapping_url"])
+        schema = load_schema_from_url(context["params"]["schema_url"])
+
+        transformed_items, validation_errors = transform_with_jsonata(
+            data["items"], mapping_expr, schema
+        )
+
+        if validation_errors:
+            raise ValueError(f"Found {len(validation_errors)} validation errors.")
+
+        return transformed_items
+
+    @task
+    def create_datasets(
+        transformed_items: List[Dict[str, Any]], **context
+    ) -> DatasetCollection:
+        """Create Synapse datasets and collection.
+
+        This task:
+        1. Creates a new Dataset Collection with predefined columns
+        2. For each transformed item:
+           - Creates a new Dataset with the item's title and description
+           - Adds the Dataset to the collection
+        3. Stores the collection in Synapse
+
+        Args:
+            transformed_items: List of transformed and validated items
+            **context: Airflow task context containing DAG parameters
+
+        Returns:
+            DatasetCollection: The created and stored dataset collection
+        """
+        syn_hook = SynapseHook(context["params"]["synapse_conn_id"])
+        synapse_client = syn_hook.client
+
+        columns = [
+            Column(name="id", column_type=ColumnType.ENTITYID),
+            Column(name="title", column_type=ColumnType.STRING, maximum_size=200),
+            Column(name="creator", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="keywords", column_type=ColumnType.STRING, maximum_size=250),
+            Column(name="subject", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="collection", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="publisher", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="species", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="sameAs", column_type=ColumnType.STRING, maximum_size=100),
+        ]
+
+        dataset_collection = DatasetCollection(
+            name=context["params"]["collection_name"],
+            description=context["params"]["collection_description"],
+            parent_id=context["params"]["project_id"],
+            include_default_columns=True,
+            columns=columns,
+        )
+
+        for item in transformed_items:
+            dataset_description = (
+                item["description"][:1000]
+                if len(item["description"]) > 1000
+                else item["description"]
+            )
+            dataset = Dataset(
+                name=item["title"],
+                description=dataset_description,
+                parent_id=context["params"]["project_id"],
+            ).store(synapse_client=synapse_client)
+            dataset_collection.add_item(dataset)
+
+        dataset_collection = dataset_collection.store(synapse_client=synapse_client)
+        return dataset_collection
+
+    @task
+    def update_annotations(
+        dataset_collection: DatasetCollection,
+        transformed_items: List[Dict[str, Any]],
+        **context,
+    ) -> None:
+        """Update dataset annotations and create snapshot if changes are detected.
+
+        This task:
+        1. Queries the current state of the dataset collection
+        2. Prepares new annotation data from the transformed items
+        3. Updates the collection with the new annotations
+        4. Queries the updated state
+        5. Compares the before and after states
+        6. If changes are detected:
+           - Creates a summary of which columns were modified
+           - Creates a new snapshot with a descriptive comment
+           - The comment includes timestamp, changed columns, and dataset count
+
+        Args:
+            dataset_collection: The dataset collection to update
+            transformed_items: List of transformed items containing new annotations
+            **context: Airflow task context containing DAG parameters
+        """
+        syn_hook = SynapseHook(context["params"]["synapse_conn_id"])
+        synapse_client = syn_hook.client
+
+        dataset_ids = [item.id for item in dataset_collection.items]
+
+        # Get current data before update
+        current_data = dataset_collection.query(
+            query=f"SELECT * from {dataset_collection.id}",
+            synapse_client=synapse_client,
+        )
+        current_df = pd.DataFrame(current_data)
path: dags/als-kp-dataset-dag.py
user: login: BryanFauble
body: Iirc the query function is already returning a DataFrame. Is this extra line needed? Could instead on the line above set the type explicitly like:

`current_data: pd.DataFrame = dataset_collection.query(`

created_at: 2025-05-08T18:59:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,338 @@
+"""ALS Knowledge Portal Dataset Collection DAG
+
+This DAG automates the process of creating and maintaining a Synapse Dataset Collection
+for the ALS Knowledge Portal. The DAG follows these steps:
+
+1. Fetch data from the C-Path API using an authentication token stored in Airflow Variables
+2. Transform the raw data using a JSONata mapping expression and validate against a JSON Schema
+3. Create or update Synapse Datasets for each item in the transformed data
+4. Create or update a Dataset Collection containing all the datasets
+5. Create or update annotations for each dataset in the collection
+6. Create a new snapshot of the collection if any changes were detected
+
+The DAG runs monthly and uses Airflow Variables for configuration.
+"""
+
+from datetime import datetime
+import requests
+from typing import Dict, List, Tuple, Any, Optional
+
+import pandas as pd
+from jsonata import jsonata
+from jsonschema import validate, ValidationError
+
+from synapseclient.models import Dataset, DatasetCollection, Column, ColumnType
+from orca.services.synapse import SynapseHook
+
+from airflow.decorators import task, dag
+from airflow.models import Variable, Param
+
+
+dag_params = {
+    "project_id": Param("syn64892175", type="string"),
+    "mapping_url": Param(
+        "https://raw.githubusercontent.com/amp-als/data-model/refs/heads/main/mapping/cpath.jsonata",
+        type="string",
+    ),
+    "schema_url": Param(
+        "https://raw.githubusercontent.com/amp-als/data-model/refs/heads/main/json-schemas/Dataset.json",
+        type="string",
+    ),
+    "cpath_api_url": Param(
+        "https://fair.dap.c-path.org/api/collections/als-kp/datasets", type="string"
+    ),
+    "collection_name": Param("Dataset collection (Production)", type="string"),
+    "collection_description": Param(
+        "A collection of datasets curated for the ALS Knowledge Portal", type="string"
+    ),
+    "synapse_conn_id": Param("SYNAPSE_ORCA_SERVICE_ACCOUNT_CONN", type="string"),
+}
+
+dag_config = {
+    "schedule_interval": "0 0 1 * *",  # Run on the first day of the month at midnight
+    "start_date": datetime(2025, 5, 1),
+    "catchup": False,
+    "default_args": {
+        "retries": 2,
+    },
+    "tags": ["als-kp"],
+    "params": dag_params,
+}
+
+
+def load_mapping_from_url(url: str) -> str:
+    """Load the JSONata mapping expression from a URL"""
+    response = requests.get(url)
+    response.raise_for_status()
+    return response.text
+
+
+def load_schema_from_url(url: str) -> Dict[str, Any]:
+    """Load the JSON Schema from a URL"""
+    response = requests.get(url)
+    response.raise_for_status()
+    return response.json()
+
+
+def validate_item(
+    item: Dict[str, Any], schema: Dict[str, Any]
+) -> Tuple[bool, Optional[str]]:
+    """Validate an item against a JSON Schema"""
+    try:
+        validate(instance=item, schema=schema)
+        return True, None
+    except ValidationError as e:
+        return False, str(e)
+
+
+def transform_with_jsonata(
+    source_items: List[Dict[str, Any]],
+    mapping_expr: str,
+    schema: Optional[Dict[str, Any]] = None,
+) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
+    """Transform a list of items using a JSONata expression and validate against schema"""
+    expr = jsonata.Jsonata(mapping_expr)
+    transformed_items: List[Dict[str, Any]] = []
+    validation_errors: List[Dict[str, Any]] = []
+
+    for i, item in enumerate(source_items):
+        result = expr.evaluate(item)
+        if schema:
+            is_valid, error = validate_item(result, schema)
+            if not is_valid:
+                validation_errors.append(
+                    {"item_index": i, "error": error, "transformed_item": result}
+                )
+                continue
+        transformed_items.append(result)
+
+    return transformed_items, validation_errors
+
+
+@dag(**dag_config)
+def als_kp_dataset_dag():
+    @task
+    def fetch_cpath_data(**context) -> Dict[str, Any]:
+        """Fetch data from C-Path API using auth token from Airflow Variables.
+
+        Args:
+            **context: Airflow task context containing DAG parameters
+
+        Returns:
+            Dict[str, Any]: JSON response from the C-Path API containing dataset items
+
+        Raises:
+            requests.exceptions.RequestException: If the API request fails
+        """
+        headers = {
+            "accept": "application/json",
+            "Authorization": f"Bearer {Variable.get('CPATH_API_TOKEN')}",
+        }
+
+        response = requests.get(context["params"]["cpath_api_url"], headers=headers)
+        response.raise_for_status()
+        return response.json()
+
+    @task
+    def transform_data(data: Dict[str, Any], **context) -> List[Dict[str, Any]]:
+        """Transform the data using JSONata mapping and validate against schema.
+
+        This task:
+        1. Loads the JSONata mapping expression from the specified URL
+        2. Loads the JSON Schema from the specified URL
+        3. Applies the mapping to each item in the input data
+        4. Validates each transformed item against the schema
+        5. Returns only the valid transformed items
+
+        Args:
+            data: Raw data from the C-Path API
+            **context: Airflow task context containing DAG parameters
+
+        Returns:
+            List[Dict[str, Any]]: List of transformed and validated items
+
+        Raises:
+            ValueError: If any validation errors are found
+            requests.exceptions.RequestException: If loading mapping or schema fails
+        """
+        mapping_expr = load_mapping_from_url(context["params"]["mapping_url"])
+        schema = load_schema_from_url(context["params"]["schema_url"])
+
+        transformed_items, validation_errors = transform_with_jsonata(
+            data["items"], mapping_expr, schema
+        )
+
+        if validation_errors:
+            raise ValueError(f"Found {len(validation_errors)} validation errors.")
+
+        return transformed_items
+
+    @task
+    def create_datasets(
+        transformed_items: List[Dict[str, Any]], **context
+    ) -> DatasetCollection:
+        """Create Synapse datasets and collection.
+
+        This task:
+        1. Creates a new Dataset Collection with predefined columns
+        2. For each transformed item:
+           - Creates a new Dataset with the item's title and description
+           - Adds the Dataset to the collection
+        3. Stores the collection in Synapse
+
+        Args:
+            transformed_items: List of transformed and validated items
+            **context: Airflow task context containing DAG parameters
+
+        Returns:
+            DatasetCollection: The created and stored dataset collection
+        """
+        syn_hook = SynapseHook(context["params"]["synapse_conn_id"])
+        synapse_client = syn_hook.client
+
+        columns = [
+            Column(name="id", column_type=ColumnType.ENTITYID),
+            Column(name="title", column_type=ColumnType.STRING, maximum_size=200),
+            Column(name="creator", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="keywords", column_type=ColumnType.STRING, maximum_size=250),
+            Column(name="subject", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="collection", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="publisher", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="species", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="sameAs", column_type=ColumnType.STRING, maximum_size=100),
+        ]
+
+        dataset_collection = DatasetCollection(
+            name=context["params"]["collection_name"],
+            description=context["params"]["collection_description"],
+            parent_id=context["params"]["project_id"],
+            include_default_columns=True,
+            columns=columns,
+        )
+
+        for item in transformed_items:
+            dataset_description = (
+                item["description"][:1000]
+                if len(item["description"]) > 1000
+                else item["description"]
+            )
+            dataset = Dataset(
+                name=item["title"],
+                description=dataset_description,
+                parent_id=context["params"]["project_id"],
+            ).store(synapse_client=synapse_client)
+            dataset_collection.add_item(dataset)
+
+        dataset_collection = dataset_collection.store(synapse_client=synapse_client)
+        return dataset_collection
+
+    @task
+    def update_annotations(
+        dataset_collection: DatasetCollection,
+        transformed_items: List[Dict[str, Any]],
+        **context,
+    ) -> None:
+        """Update dataset annotations and create snapshot if changes are detected.
+
+        This task:
+        1. Queries the current state of the dataset collection
+        2. Prepares new annotation data from the transformed items
+        3. Updates the collection with the new annotations
+        4. Queries the updated state
+        5. Compares the before and after states
+        6. If changes are detected:
+           - Creates a summary of which columns were modified
+           - Creates a new snapshot with a descriptive comment
+           - The comment includes timestamp, changed columns, and dataset count
+
+        Args:
+            dataset_collection: The dataset collection to update
+            transformed_items: List of transformed items containing new annotations
+            **context: Airflow task context containing DAG parameters
+        """
+        syn_hook = SynapseHook(context["params"]["synapse_conn_id"])
+        synapse_client = syn_hook.client
+
+        dataset_ids = [item.id for item in dataset_collection.items]
+
+        # Get current data before update
+        current_data = dataset_collection.query(
+            query=f"SELECT * from {dataset_collection.id}",
+            synapse_client=synapse_client,
+        )
+        current_df = pd.DataFrame(current_data)
path: dags/als-kp-dataset-dag.py
user: login: BWMac
body: Ah yeah that's right, thanks
created_at: 2025-05-08T19:01:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,338 @@
+"""ALS Knowledge Portal Dataset Collection DAG
+
+This DAG automates the process of creating and maintaining a Synapse Dataset Collection
+for the ALS Knowledge Portal. The DAG follows these steps:
+
+1. Fetch data from the C-Path API using an authentication token stored in Airflow Variables
+2. Transform the raw data using a JSONata mapping expression and validate against a JSON Schema
+3. Create or update Synapse Datasets for each item in the transformed data
+4. Create or update a Dataset Collection containing all the datasets
+5. Create or update annotations for each dataset in the collection
+6. Create a new snapshot of the collection if any changes were detected
+
+The DAG runs monthly and uses Airflow Variables for configuration.
+"""
+
+from datetime import datetime
+import requests
+from typing import Dict, List, Tuple, Any, Optional
+
+import pandas as pd
+from jsonata import jsonata
+from jsonschema import validate, ValidationError
+
+from synapseclient.models import Dataset, DatasetCollection, Column, ColumnType
+from orca.services.synapse import SynapseHook
+
+from airflow.decorators import task, dag
+from airflow.models import Variable, Param
+
+
+dag_params = {
+    "project_id": Param("syn64892175", type="string"),
+    "mapping_url": Param(
+        "https://raw.githubusercontent.com/amp-als/data-model/refs/heads/main/mapping/cpath.jsonata",
+        type="string",
+    ),
+    "schema_url": Param(
+        "https://raw.githubusercontent.com/amp-als/data-model/refs/heads/main/json-schemas/Dataset.json",
+        type="string",
+    ),
+    "cpath_api_url": Param(
+        "https://fair.dap.c-path.org/api/collections/als-kp/datasets", type="string"
+    ),
+    "collection_name": Param("Dataset collection (Production)", type="string"),
+    "collection_description": Param(
+        "A collection of datasets curated for the ALS Knowledge Portal", type="string"
+    ),
+    "synapse_conn_id": Param("SYNAPSE_ORCA_SERVICE_ACCOUNT_CONN", type="string"),
+}
+
+dag_config = {
+    "schedule_interval": "0 0 1 * *",  # Run on the first day of the month at midnight
+    "start_date": datetime(2025, 5, 1),
+    "catchup": False,
+    "default_args": {
+        "retries": 2,
+    },
+    "tags": ["als-kp"],
+    "params": dag_params,
+}
+
+
+def load_mapping_from_url(url: str) -> str:
+    """Load the JSONata mapping expression from a URL"""
+    response = requests.get(url)
+    response.raise_for_status()
+    return response.text
+
+
+def load_schema_from_url(url: str) -> Dict[str, Any]:
+    """Load the JSON Schema from a URL"""
+    response = requests.get(url)
+    response.raise_for_status()
+    return response.json()
+
+
+def validate_item(
+    item: Dict[str, Any], schema: Dict[str, Any]
+) -> Tuple[bool, Optional[str]]:
+    """Validate an item against a JSON Schema"""
+    try:
+        validate(instance=item, schema=schema)
+        return True, None
+    except ValidationError as e:
+        return False, str(e)
+
+
+def transform_with_jsonata(
+    source_items: List[Dict[str, Any]],
+    mapping_expr: str,
+    schema: Optional[Dict[str, Any]] = None,
+) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
+    """Transform a list of items using a JSONata expression and validate against schema"""
+    expr = jsonata.Jsonata(mapping_expr)
+    transformed_items: List[Dict[str, Any]] = []
+    validation_errors: List[Dict[str, Any]] = []
+
+    for i, item in enumerate(source_items):
+        result = expr.evaluate(item)
+        if schema:
+            is_valid, error = validate_item(result, schema)
+            if not is_valid:
+                validation_errors.append(
+                    {"item_index": i, "error": error, "transformed_item": result}
+                )
+                continue
+        transformed_items.append(result)
+
+    return transformed_items, validation_errors
+
+
+@dag(**dag_config)
+def als_kp_dataset_dag():
+    @task
+    def fetch_cpath_data(**context) -> Dict[str, Any]:
+        """Fetch data from C-Path API using auth token from Airflow Variables.
+
+        Args:
+            **context: Airflow task context containing DAG parameters
+
+        Returns:
+            Dict[str, Any]: JSON response from the C-Path API containing dataset items
+
+        Raises:
+            requests.exceptions.RequestException: If the API request fails
+        """
+        headers = {
+            "accept": "application/json",
+            "Authorization": f"Bearer {Variable.get('CPATH_API_TOKEN')}",
+        }
+
+        response = requests.get(context["params"]["cpath_api_url"], headers=headers)
+        response.raise_for_status()
+        return response.json()
+
+    @task
+    def transform_data(data: Dict[str, Any], **context) -> List[Dict[str, Any]]:
+        """Transform the data using JSONata mapping and validate against schema.
+
+        This task:
+        1. Loads the JSONata mapping expression from the specified URL
+        2. Loads the JSON Schema from the specified URL
+        3. Applies the mapping to each item in the input data
+        4. Validates each transformed item against the schema
+        5. Returns only the valid transformed items
+
+        Args:
+            data: Raw data from the C-Path API
+            **context: Airflow task context containing DAG parameters
+
+        Returns:
+            List[Dict[str, Any]]: List of transformed and validated items
+
+        Raises:
+            ValueError: If any validation errors are found
+            requests.exceptions.RequestException: If loading mapping or schema fails
+        """
+        mapping_expr = load_mapping_from_url(context["params"]["mapping_url"])
+        schema = load_schema_from_url(context["params"]["schema_url"])
+
+        transformed_items, validation_errors = transform_with_jsonata(
+            data["items"], mapping_expr, schema
+        )
+
+        if validation_errors:
+            raise ValueError(f"Found {len(validation_errors)} validation errors.")
+
+        return transformed_items
+
+    @task
+    def create_datasets(
+        transformed_items: List[Dict[str, Any]], **context
+    ) -> DatasetCollection:
+        """Create Synapse datasets and collection.
+
+        This task:
+        1. Creates a new Dataset Collection with predefined columns
+        2. For each transformed item:
+           - Creates a new Dataset with the item's title and description
+           - Adds the Dataset to the collection
+        3. Stores the collection in Synapse
+
+        Args:
+            transformed_items: List of transformed and validated items
+            **context: Airflow task context containing DAG parameters
+
+        Returns:
+            DatasetCollection: The created and stored dataset collection
+        """
+        syn_hook = SynapseHook(context["params"]["synapse_conn_id"])
+        synapse_client = syn_hook.client
+
+        columns = [
+            Column(name="id", column_type=ColumnType.ENTITYID),
+            Column(name="title", column_type=ColumnType.STRING, maximum_size=200),
+            Column(name="creator", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="keywords", column_type=ColumnType.STRING, maximum_size=250),
+            Column(name="subject", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="collection", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="publisher", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="species", column_type=ColumnType.STRING, maximum_size=100),
+            Column(name="sameAs", column_type=ColumnType.STRING, maximum_size=100),
+        ]
+
+        dataset_collection = DatasetCollection(
+            name=context["params"]["collection_name"],
+            description=context["params"]["collection_description"],
+            parent_id=context["params"]["project_id"],
+            include_default_columns=True,
+            columns=columns,
+        )
+
+        for item in transformed_items:
+            dataset_description = (
+                item["description"][:1000]
+                if len(item["description"]) > 1000
+                else item["description"]
+            )
+            dataset = Dataset(
+                name=item["title"],
+                description=dataset_description,
+                parent_id=context["params"]["project_id"],
+            ).store(synapse_client=synapse_client)
+            dataset_collection.add_item(dataset)
+
+        dataset_collection = dataset_collection.store(synapse_client=synapse_client)
+        return dataset_collection
+
+    @task
+    def update_annotations(
+        dataset_collection: DatasetCollection,
+        transformed_items: List[Dict[str, Any]],
+        **context,
+    ) -> None:
+        """Update dataset annotations and create snapshot if changes are detected.
+
+        This task:
+        1. Queries the current state of the dataset collection
+        2. Prepares new annotation data from the transformed items
+        3. Updates the collection with the new annotations
+        4. Queries the updated state
+        5. Compares the before and after states
+        6. If changes are detected:
+           - Creates a summary of which columns were modified
+           - Creates a new snapshot with a descriptive comment
+           - The comment includes timestamp, changed columns, and dataset count
+
+        Args:
+            dataset_collection: The dataset collection to update
+            transformed_items: List of transformed items containing new annotations
+            **context: Airflow task context containing DAG parameters
+        """
+        syn_hook = SynapseHook(context["params"]["synapse_conn_id"])
+        synapse_client = syn_hook.client
+
+        dataset_ids = [item.id for item in dataset_collection.items]
+
+        # Get current data before update
+        current_data = dataset_collection.query(
+            query=f"SELECT * from {dataset_collection.id}",
+            synapse_client=synapse_client,
+        )
+        current_df = pd.DataFrame(current_data)
+
+        # Prepare and apply new data
+        annotation_data = pd.DataFrame(
+            {
+                "id": dataset_ids,
+                **{
+                    key: [
+                        (
+                            ", ".join(item[key])
+                            if isinstance(item[key], list)
+                            else item[key]
+                        )
+                        for item in transformed_items
+                    ]
+                    for key in [
+                        "title",
+                        "creator",
+                        "keywords",
+                        "subject",
+                        "collection",
+                        "publisher",
+                        "species",
+                        "sameAs",
+                    ]
+                },
+            }
+        )
+
+        # Update the rows
+        dataset_collection.update_rows(
path: dags/als-kp-dataset-dag.py
user: login: BryanFauble
body: I think you'll want to set this to true

https://github.com/Sage-Bionetworks/synapsePythonClient/blob/70e53e692673930a7c3ce7882aec5faa94632bad/synapseclient/models/dataset.py#L228
created_at: 2025-05-08T19:02:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,387 @@
+"""ALS Knowledge Portal Dataset Collection DAG
+
+This DAG automates the process of creating and maintaining a Synapse Dataset Collection
+for the ALS Knowledge Portal. The DAG follows these steps:
+
+1. Fetch data from the C-Path API using an authentication token stored in Airflow Variables
+2. Transform the raw data using a JSONata mapping expression and validate against a JSON Schema
+3. Create or update Synapse Datasets for each item in the transformed data
+4. Create or update a Dataset Collection containing all the datasets
+5. Create or update annotations for each dataset in the collection
+6. Create a new snapshot of the collection if any changes were detected
+
+The DAG runs monthly and uses Airflow Variables for configuration.
+"""
+
+from datetime import datetime
+import requests
+from typing import Dict, List, Tuple, Any, Optional
+
+import pandas as pd
+from jsonata import jsonata
+from jsonschema import validate, ValidationError
+
+from synapseclient.models import Dataset, DatasetCollection, Column, ColumnType
+from orca.services.synapse import SynapseHook
+
+from airflow.decorators import task, dag
+from airflow.models import Variable, Param
+
+
+dag_params = {
+    "project_id": Param("syn64892175", type="string"),
+    "mapping_url": Param(
+        "https://raw.githubusercontent.com/amp-als/data-model/refs/heads/main/mapping/cpath.jsonata",
path: dags/als-kp-dataset-dag.py
user: login: thomasyu888
body: 🔥 Is mapping to main a risk here?

Great work here! Thanks!
created_at: 2025-05-08T19:10:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,387 @@
+"""ALS Knowledge Portal Dataset Collection DAG
+
+This DAG automates the process of creating and maintaining a Synapse Dataset Collection
+for the ALS Knowledge Portal. The DAG follows these steps:
+
+1. Fetch data from the C-Path API using an authentication token stored in Airflow Variables
+2. Transform the raw data using a JSONata mapping expression and validate against a JSON Schema
+3. Create or update Synapse Datasets for each item in the transformed data
+4. Create or update a Dataset Collection containing all the datasets
+5. Create or update annotations for each dataset in the collection
+6. Create a new snapshot of the collection if any changes were detected
+
+The DAG runs monthly and uses Airflow Variables for configuration.
+"""
+
+from datetime import datetime
+import requests
+from typing import Dict, List, Tuple, Any, Optional
+
+import pandas as pd
+from jsonata import jsonata
+from jsonschema import validate, ValidationError
+
+from synapseclient.models import Dataset, DatasetCollection, Column, ColumnType
+from orca.services.synapse import SynapseHook
+
+from airflow.decorators import task, dag
+from airflow.models import Variable, Param
+
+
+dag_params = {
+    "project_id": Param("syn64892175", type="string"),
+    "mapping_url": Param(
+        "https://raw.githubusercontent.com/amp-als/data-model/refs/heads/main/mapping/cpath.jsonata",
path: dags/als-kp-dataset-dag.py
user: login: BWMac
body: Good point, I can map to a specific commit hash instead
created_at: 2025-05-08T19:13:17Z
author_association: CONTRIBUTOR

PR comments:
user: login: sonarqubecloud[bot]
created_at: 2025-05-08T19:16:00Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=Sage-Bionetworks-Workflows_orca-recipes&pullRequest=89) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [5 New issues](https://sonarcloud.io/project/issues?id=Sage-Bionetworks-Workflows_orca-recipes&pullRequest=89&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=Sage-Bionetworks-Workflows_orca-recipes&pullRequest=89&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=Sage-Bionetworks-Workflows_orca-recipes&pullRequest=89&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=Sage-Bionetworks-Workflows_orca-recipes&pullRequest=89&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=Sage-Bionetworks-Workflows_orca-recipes&pullRequest=89&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=Sage-Bionetworks-Workflows_orca-recipes&pullRequest=89)


author_association: NONE
user: login: BWMac
created_at: 2025-05-12T14:46:09Z
body: The output of the DAG has been [approved](https://sagebionetworks.jira.com/browse/DPE-1288?focusedCommentId=249427). Merging now and unpausing in production.
author_association: CONTRIBUTOR