Repository: Sage-Bionetworks-Workflows/orca-recipes
Pull request #83: [DPE-1275] Move to Synapse API, Automate deletion of S3 objects/Synapse Rows, File/Dataset versioning corrections
Author: login: BryanFauble
Created: 2025-04-02T23:53:18Z | Merged: 2025-04-09T20:59:44Z

# **Problem:**

- We were using the Snowflake datawarehouse to source information in order to build this process. Per [these guidelines](https://sagebionetworks.jira.com/wiki/spaces/DPE/pages/4043079690/Data+Warehouse+Usage+Guidelines+policies) we want to limit usage of the warehouse.
- Deleting S3 objects that were no longer present in the dataset collections was a manual process
- We did not capture version information for the file entities
- We did not persist S3 object links back to a Synapse table to use for downstream processes

# **Solution:**

- Moving all (Except MD5 retrieval) reads to occur via unauthenticated/anonymous Synapse APIs
- Adding logic to delete out S3 objects which are no longer present in the dataset collections given.
- Adding version information for the croissant file
- Adding support to push s3 object links to a Synapse table
- Support deleting rows out of the Synapse table which should no longer show

# **Testing:**

- I verified that I could run this DAG in github codespaces with all params set to True
- Data is pushed to this (INTERNAL TO SAGE FOR NOW) table: https://www.synapse.org/Synapse:syn65903895/tables/


TODO: 

- [x] Naming convention changes for the croissant spec file are ok
- [x] The JSON LD file contains all expected changes
- [x] Push the links to the S3 objects back into a Synapse table that has columns for: Synapse ID of the Dataset collection, Synapse ID of the Dataset, Version of the Dataset, Link to the S3 object



Diff:
diff --git a/dags/synapse-dataset-to-croissant.py b/dags/synapse-dataset-to-croissant.py
index 05a1ef9d..bf48ca14 100644
--- a/dags/synapse-dataset-to-croissant.py
+++ b/dags/synapse-dataset-to-croissant.py
@@ -1,7 +1,23 @@
 """
-This DAG queries snowflake for the given dataset collections to look for datasets.
+This DAG interacts with a few different services to accomplish the following:
 
-For each dataset in the dataset collections, it queries the dataset in Snowflake and 
+- (Synapse, Unauthenticated) Given list of dataset collections query Synapse to retrieve Anonymously accessible datasets
+- (Synapse, Unauthenticated) For each dataset in the dataset collections, query Synapse Anonymously to retrieve metadata about each dataset
+- (Synapse, Unauthenticated) For each Anonymously accessible dataset, query Synapse Anonymously to retrieve the files attached to the dataset
+- (Snowflake, Authenticated) For each file which is not Anonymously downloadable, query Snowflake to retrieve the content_md5 (Authenticated calls to the datawarehouse)
+- (S3, Authenticated) For each dataset push an object a public S3 bucket in the `org-sagebase-dpe-prod` AWS account
+- (Synapse, Authenticated) For each dataset query a Synapse table which contains links to the S3 object
+- (Synapse, Authenticated) Delete rows from the Synapse table which are not in a dataset collection
+- (Synapse, Authenticated) For each dataset push a link for the S3 object to a Synapse table via the authenticated Synapse client
+- (S3, Authenticated) Delete S3 objects from the S3 bucket which are not in a dataset collection
+
+
+To add more datasets to be processed by this dag:
+Simply add the dataset collections to the `dataset_collections` parameter in the DAG to
+run this process for the given dataset collections.
+
+
+For each dataset in the dataset collections, it queries the dataset in Synapse and
 pushes the results to a Croissant file in S3. The S3 bucket is stored in the
 `org-sagebase-dpe-prod` AWS account. That S3 bucket is deployed to AWS through the code
 in this PR: https://github.com/Sage-Bionetworks-Workflows/eks-stack/pull/57 .
@@ -9,35 +25,33 @@
 Additional note on the pushing to S3. The way that the S3 hook is set up is that it will
 log in as an AWS user to accomplish the required work. In order for this DAG to run the
 user that is running the DAG must have the correct permissions to access the S3 bucket
-and write to it. This was set up on the `airflow-secrets-backend` user in the 
+and write to it. This was set up on the `airflow-secrets-backend` user in the
 `org-sagebase-dpe-prod` AWS account. The user has an inline policy to grant PutObject,
-GetObject, and ListBucket permissions to the S3 bucket.
-
-Simply add the dataset collections to the `dataset_collections` parameter in the DAG to
-run this process for the given dataset collections.
+GetObject, DeleteObject, and ListBucket permissions to the S3 bucket.
 
 In addition this DAG has been set up with the first iteration of the OpenTelemetry
 integration for Apache Airflow. OTEL support is officially added in 2.10.0, but requires
-that we also upgrade the production Airflow server.
+that we also set up the production Airflow server to support OTEL.
 
 DAG Parameters:
-- `snowflake_developer_service_conn`: A JSON-formatted string containing the connection details required to authenticate and connect to Snowflake.
-- `synapse_conn_id`: The connection ID for the Synapse connection.
-- `dataset_collections`: The dataset collections to query for datasets.
-- `push_results_to_s3`: A boolean to indicate if the results should be pushed to S3.
-                        When set to `False`, the results will be printed to the logs.
-- `aws_conn_id`: The connection ID for the AWS connection. Used to authenticate with S3.
+- Review the DAG Parameters under the `@dag` decorated function
 """
 
+import json
 import os
+import re
+import tempfile
+import mimetypes
 from datetime import datetime
 from io import BytesIO
 from logging import NOTSET, Logger, getLogger
-from typing import Dict, List
+from types import MethodType
+from typing import Any, Dict, List
+from urllib.parse import quote_plus
 
 from airflow.decorators import dag, task
-from airflow.models.param import Param
 from airflow.models import Variable
+from airflow.models.param import Param
 from airflow.providers.amazon.aws.hooks.s3 import S3Hook
 from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
 from opentelemetry import context as otel_context
@@ -55,12 +69,24 @@
 from opentelemetry.sdk.trace.export import SimpleSpanProcessor
 from opentelemetry.trace.propagation.tracecontext import \
     TraceContextTextMapPropagator
+from orca.services.synapse import SynapseHook
+from pandas import DataFrame
+from synapseclient import Entity, Synapse, Table
+from synapseclient.core.exceptions import (SynapseAuthenticationError,
+                                           SynapseHTTPError)
+from synapseclient.core.retry import with_retry
+from synapseclient.models import File
+from synapseclient.core.utils import delete_none_keys
 
 dag_params = {
     "snowflake_developer_service_conn": Param("SNOWFLAKE_DEVELOPER_SERVICE_RAW_CONN", type="string"),
     "synapse_conn_id": Param("SYNAPSE_ORCA_SERVICE_ACCOUNT_CONN", type="string"),
     "dataset_collections": Param(["syn50913342"], type="array"),
     "push_results_to_s3": Param(True, type="boolean"),
+    "push_links_to_synapse": Param(True, type="boolean"),
+    "delete_out_of_date_from_s3": Param(True, type="boolean"),
+    "delete_out_of_date_from_synapse": Param(True, type="boolean"),
+    "dataset_collections_for_cleanup": Param(["syn50913342"], type="array"),
     "aws_conn_id": Param("AWS_SYNAPSE_CROISSANT_METADATA_S3_CONN", type="string"),
 }
 
@@ -89,6 +115,14 @@
 MY_DEPLOYMENT_ENVIRONMENT = "prod"
 # MY_DEPLOYMENT_ENVIRONMENT = "local"
 
+SYNAPSE_TABLE_FOR_CROISSANT_LINKS = "syn65903895"
+
+# Regular expression to match `_{dataset_id}.{dataset_version}_`
+DATASET_ID_VERSION_PATTERN = r"_syn\d+\.\d+_"
+
+# Regular expression to match `_datasetCollection_{dataset_collection}_`
+DATASET_COLLECTION_PATTERN = r"_datasetCollection_syn\d+_"
+
 
 def set_up_tracing() -> Tracer:
     """
@@ -100,9 +134,9 @@ def set_up_tracing() -> Tracer:
     - MY_SERVICE_VERSION: The version of the service.
 
 
-    These attributes are used by the OTLP exporter to tag the telemetry data. They 
+    These attributes are used by the OTLP exporter to tag the telemetry data. They
     should be set to something that uniquely identifies the code that is producing this
-    data. Within the telemetry backend these attributes will be used to filter and 
+    data. Within the telemetry backend these attributes will be used to filter and
     group the data.
     """
     service_name = os.environ.get("SERVICE_NAME", MY_SERVICE_NAME)
@@ -138,9 +172,9 @@ def set_up_logging() -> Logger:
     - MY_SERVICE_VERSION: The version of the service.
 
 
-    These attributes are used by the OTLP exporter to tag the telemetry data. They 
+    These attributes are used by the OTLP exporter to tag the telemetry data. They
     should be set to something that uniquely identifies the code that is producing this
-    data. Within the telemetry backend these attributes will be used to filter and 
+    data. Within the telemetry backend these attributes will be used to filter and
     group the data.
     """
     service_name = os.environ.get("SERVICE_NAME", MY_SERVICE_NAME)
@@ -165,17 +199,561 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        modified_file_id = int(file.id.replace("syn", ""))
+        ids_of_files.append(modified_file_id)
+        id_and_version_pairs.append(modified_file_id)
+        id_and_version_pairs.append(file.version_number)
+        files_to_find_md5_in_snowflake[int(file.id.replace(
+            "syn", ""))] = file.version_number
+
+    file_md5_and_types = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5,
+                flattened.value:fileHandleId::int AS fileHandleId
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5,
+            vd.fileHandleId,
+            COALESCE(fl.content_type, 'NOT_SET') as CONTENT_TYPE,
+        FROM version_data AS vd
+        LEFT JOIN synapse_data_warehouse.synapse.file_latest fl
+            ON fl.id = vd.fileHandleId
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5_and_types = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file_md5_and_types.empty and int(file.id.replace("syn", "")) in file_md5_and_types["ID"].values:
+            file_md5 = file_md5_and_types.loc[file_md5_and_types["ID"] == int(
+                file.id.replace("syn", "")), "CONTENTMD5"].values[0]
+            
+            file_content_type = file_md5_and_types.loc[file_md5_and_types["ID"] == int(
+                file.id.replace("syn", "")), "CONTENT_TYPE"].values[0]
+        else:
+            file_md5 = "unknown_md5"
+
+        if not file_content_type or file_content_type == "NOT_SET":
+            if file.file_handle and file.file_handle.file_name:
+                file_content_type = mimetypes.guess_type(
+                    file.file_handle.file_name, strict=False)[0]
+            else:
+                file_content_type = mimetypes.guess_type(
+                    file.name, strict=False)[0]
+                
+            if not file_content_type:
+                # For binary documents without a specific or known subtype, application/octet-stream should be used.
+                # Source: https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/MIME_types
+                file_content_type = "application/octet-stream"
+
+        distribution_files.append(
+            {
+                "@type": "FileObject",
+                "@id": f"{file.id}.{file.version_number}",
+                "name": f"{file.name}",
+                "description": file.description if file.description else f"Data file associated with {file.name}",
+                "contentUrl": f"https://www.synapse.org/Synapse:{file.id}.{file.version_number}",
+                "encodingFormat": file_content_type,
+                "md5": file_md5,
+                "sha256": "unknown",
+            }
+        )
+    return distribution_files
+
+
+def construct_record_set_section_for_files(files_attached_to_dataset: List[File]) -> Dict[str, str]:
+    """
+    Construct the record set section for the files attached to the dataset. This is used
+    to extract the keys from the annotations of the files and create a Field for each
+    key.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+
+    Returns:
+        The record set section for the files attached to the dataset.
+    """
+    unique_annotation_keys = set()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        file_annotations = file.annotations.keys()
+        for key in file_annotations:
+            unique_annotation_keys.add(key)
+
+    metadata_fields = []
+    # Sort the set of unique_annotation_keys ignoring case, but keep the case in the result
+    unique_annotation_keys = sorted(unique_annotation_keys, key=str.casefold)
+
+    for key in unique_annotation_keys:
+        metadata_fields.append(
+            {
+                "@type": "Field",
+                "@id": f"metadata/{key}",
+                "name": key,
+                "description": "",
+                "dataType": "sc:Text",
+                "source": {
+                    "fileObject": {"@id": "metadata"},
+                    "extract": {"column": key}
+                }
+            }
+        )
+
+    return {
+        "@type": "RecordSet",
+        "@id": "default",
+        "name": "default",
+        "description": "Metadata for the dataset",
+        "field": metadata_fields
+    }
+
+
+def extract_s3_objects_to_delete(bucket_objects: List[str], dataset_collections_to_consider_for_deletion: List[str], combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the S3 objects to delete from the S3 bucket. This is done by retrieving
+    all of the objects in the S3 bucket and checking if the object matches one of the
+    datasets that will be kept after the full DAG as been run. Every object in the S3
+    bucket that matches the pattern `_syn####.###_` and
+    `_datasetCollection_syn####_` will be considered for deletion. The DAG param
+    `dataset_collections_for_cleanup` is used to filter the objects that will be
+    considered for deletion as the second filter.
+
+    The last filter enforces that each object in the S3 bucket must match a dataset
+    id, dataset version, dataset collection, and dataset name in the
+    `combined_dataset_collection_and_datasets` list. If the object does not match
+    any of the objects in the list, then it will be marked for deletion.
+
+    Arguments:
+        bucket_objects: The list of objects in the S3 bucket.
+        dataset_collections_to_consider_for_deletion: The dataset collections to
+            consider for deletion. This is passed in from the
+            `dataset_collections_for_cleanup` DAG parameter.
+        combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+            dataset collections, and dataset versions to use to determine which
+            datasets to delete.
+
+    Returns:
+        The list of objects to delete from the S3 bucket.
+    """
+    objects_to_delete = []
+
+    if not bucket_objects:
+        otel_logger.info(
+            "No objects found in S3.")
+        return objects_to_delete
+
+    for bucket_object in bucket_objects:
+        dataset_id_and_version = re.search(
+            DATASET_ID_VERSION_PATTERN, bucket_object)
+        dataset_collection = re.search(
+            DATASET_COLLECTION_PATTERN, bucket_object)
+
+        if not dataset_id_and_version or not dataset_collection:
+            otel_logger.info(
+                f"Object {bucket_object} does not match the pattern. Skipping.")
+            continue
+
+        dataset_collection = dataset_collection.group(0).replace(
+            "_datasetCollection_", "").replace("_", "")
+
+        if dataset_collection not in dataset_collections_to_consider_for_deletion:
+            otel_logger.info(
+                f"Object {bucket_object} does not match a dataset collection present in `dataset_collections_for_cleanup`. Skipping.")
+            continue
+
+        dataset_id, dataset_version = dataset_id_and_version.group(0).split(
+            ".")
+        dataset_id = dataset_id.replace("_", "")
+        dataset_version = dataset_version.replace("_", "")
+        dataset_name = bucket_object.split(dataset_id_and_version.group(0))[0]
+
+        match_found_for_object = False
+
+        for combined_dataset_collection_and_dataset in combined_dataset_collection_and_datasets:
+
+            matches_dataset_collection = dataset_collection == combined_dataset_collection_and_dataset[
+                "dataset_collection"]
+            matches_dataset_id = dataset_id == combined_dataset_collection_and_dataset[
+                "dataset_id"]
+            matches_dataset_version = dataset_version == str(
+                combined_dataset_collection_and_dataset["dataset_version"])
+            matches_dataset_name = dataset_name == combined_dataset_collection_and_dataset[
+                "dataset_name"]
+
+            if matches_dataset_collection and matches_dataset_id and matches_dataset_version and matches_dataset_name:
+                match_found_for_object = True
+                break
+        if not match_found_for_object:
+            objects_to_delete.append(bucket_object)
+
+    return objects_to_delete
+
+
+def extract_synapse_rows_to_delete(synapse_rows: DataFrame, combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the rows to delete from the Synapse table. This is done by retrieving the
+    dataset ids along with their versions from the dataset collections passed in via
+    the `dataset_collections_for_cleanup` DAG parameter. The rows to delete are
+    determined by checking if the dataset id and version are present in the
+    `combined_dataset_collection_and_datasets` list. If the dataset id and version
+    are not present in the list, then the row is marked for deletion.
+
+    Arguments:
+        synapse_rows: A Pandas DataFrame with the columns `dataset`, `dataset_version`,
+            `dataset_collection`, and `ROW_ID`.
+        combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+            dataset collections, and dataset versions to use to determine which
+            datasets to delete.
+
+    Returns:
+        The list of rows to delete from the Synapse table.
+    """
+    rows_to_delete = []
+
+    if synapse_rows.empty:
+        otel_logger.info(
+            "No rows found in Synapse.")
+        return rows_to_delete
+
+    for synapse_row in synapse_rows.itertuples():
+        dataset_collection = synapse_row.dataset_collection
+        dataset_id = synapse_row.dataset
+        dataset_version = synapse_row.dataset_version
+        row_id = synapse_row.ROW_ID
+
+        match_found_for_object = False
+
+        for combined_dataset_collection_and_dataset in combined_dataset_collection_and_datasets:
+            if (dataset_collection == combined_dataset_collection_and_dataset["dataset_collection"] and
+                    dataset_id == combined_dataset_collection_and_dataset["dataset_id"] and
+                    dataset_version == combined_dataset_collection_and_dataset["dataset_version"]):
+                match_found_for_object = True
+                break
+        if not match_found_for_object:
+            rows_to_delete.append(row_id)
+
+    return rows_to_delete
+
+
+def execute_push_to_s3(dataset: Entity, dataset_id: str, dataset_version: str, s3_key: str, croissant_file: Dict[str, Any], push_to_s3: bool, **context) -> None:
+    """
+    Handle the push to S3 of the croissant file. This is done by using the S3Hook to
+    upload the file to S3. The S3 bucket is stored in the `org-sagebase-dpe-prod` AWS
+    account.
+
+    Arguments:
+        dataset: The dataset to push to S3.
+        dataset_id: The ID of the dataset.
+        dataset_version: The version of the dataset.
+        s3_key: The S3 key to use to push the file to S3.
+        croissant_file: The croissant file to push to S3.
+        push_to_s3: A boolean to indicate if the results should be pushed to S3.
+            When set to `False`, the results will be printed to the logs.
+        context: The context of the DAG run.
+
+    Returns:
+
+    """
+    try:
+        if not push_to_s3:
+            otel_logger.info(
+                f"Croissant file for [dataset: {dataset.name}, id: {dataset_id}.{dataset_version}]:\n{json.dumps(croissant_file)}")
+            return
+
+        otel_logger.info(
+            f"Uploading croissant file for [dataset: {dataset.name}, id: {dataset_id}.{dataset_version}]")
+
+        croissant_metadata_bytes = json.dumps(croissant_file).encode(
+            'utf-8')
+        metadata_file = BytesIO(croissant_metadata_bytes)
+        s3_hook = S3Hook(
+            aws_conn_id=context["params"]["aws_conn_id"], region_name=REGION_NAME, extra_args={
+                "ContentType": "application/ld+json"
+            }
+        )
+
+        otel_logger.info(
+            f"Uploading croissant file to S3: {s3_key}")
+        s3_hook.load_file_obj(file_obj=metadata_file,
+                              key=s3_key,
+                              bucket_name=BUCKET_NAME,
+                              replace=True,
+                              )
+    except Exception as ex:
+        otel_logger.exception(
+            "Failed to query snowflake and push croissant file to S3.")
+        otel_tracer.span_processor.force_flush()
+        otel_logger.handlers[0].flush()
+        raise ex
+
+
+def execute_push_to_synapse(push_to_synapse: bool, dataset: Entity, dataset_id: str, dataset_version: str, dataset_collection: str, s3_url: str, **context) -> None:
+    """
+    Handle the push to Synapse of the croissant file link. This is done by using
+    an authenticated Synapse client to first query the table to determine if an
+    update is needed. If the link already exists with the expected S3 URL, then
+    skip the update. If the link does not exist or the S3 URL is different, then
+    update the link with the new S3 URL using the authenticated Synapse client.
+
+    Arguments:
+        push_to_synapse: A boolean to indicate if the results should be pushed to
+            Synapse. When set to `False`, the results will be printed to the logs.
+        dataset: The dataset to push to Synapse.
+        dataset_id: The ID of the dataset.
+        dataset_version: The version of the dataset.
+        dataset_collection: The ID of the dataset collection where the dataset came from.
+        s3_url: The S3 URL to use for the value of the cell in the table.
+        syn_client: The unauthenticated Synapse client to use to query the table.
+        context: The context of the DAG run.
+
+    Returns:
+        None
+    """
+    try:
+        if not push_to_synapse:
+            otel_logger.info(
+                f"Croissant file link for [dataset: {dataset.name}, id: {dataset_id}.{dataset_version}]: {s3_url}")
+            return
+
+        otel_logger.info(
+            f"Uploading croissant file link to Synapse table {SYNAPSE_TABLE_FOR_CROISSANT_LINKS}"
+        )
+
+        # TODO: When 4.8.0 of the SYNPY client is released this may be
+        # replaced by the upsert functionality
+
+        # Warning: Using an authenticated Synapse Client during this section of code
+        syn_hook = SynapseHook(
+            context["params"]["synapse_conn_id"])
+        authenticated_syn_client: Synapse = syn_hook.client
+        authenticated_syn_client._rest_call = MethodType(
+            _rest_call_replacement, authenticated_syn_client)
+        existing_row = authenticated_syn_client.tableQuery(
+            query=f"SELECT * FROM {SYNAPSE_TABLE_FOR_CROISSANT_LINKS} WHERE dataset = '{dataset_id}' AND dataset_version = {dataset_version} AND dataset_collection = '{dataset_collection}'", resultsAs="csv")
+
+        try:
+            existing_row_df = existing_row.asDataFrame()
+        finally:
+            os.remove(existing_row.filepath)
+        if not existing_row_df.empty and existing_row_df["croissant_file_s3_object"].values[0] == s3_url:
+            otel_logger.info(
+                f"Croissant file link already exists in Synapse table {SYNAPSE_TABLE_FOR_CROISSANT_LINKS}. Skipping.")
+            return
+
+        df = DataFrame(
+            data={
+                "dataset": [dataset_id],
+                "dataset_version": [dataset_version],
+                "dataset_collection": [dataset_collection],
+                "croissant_file_s3_object": [s3_url]
+            }
+        )
+        temp_dir = tempfile.mkdtemp()
+        filepath = os.path.join(temp_dir, "table.csv")
+
+        schema = authenticated_syn_client.get(
+            SYNAPSE_TABLE_FOR_CROISSANT_LINKS)
+        if existing_row_df.empty:
+            # If the row does not exist, create a new row
+            authenticated_syn_client.store(
+                Table(schema=schema, values=df, filepath=filepath))
+        else:
+            # Update the existing row with the new value
+            existing_row_df["croissant_file_s3_object"] = [s3_url]
+            authenticated_syn_client.store(
+                Table(schema=schema, values=existing_row_df, filepath=filepath))
+
+        os.remove(filepath)
+    except Exception as ex:
+        otel_logger.exception(
+            "Failed to push croissant file link to Synapse.")
+        otel_tracer.span_processor.force_flush()
+        otel_logger.handlers[0].flush()
+        raise ex
+
 
 @dag(**dag_config)
 def dataset_to_croissant() -> None:
-    """Execute a query on Snowflake and report the results to a Synapse table."""
+    """Execute api calls to Synapse, and queries on snowflake to convert datasets into croissant JSON-LD files.
+
+
+    DAG Parameters:
+    - `snowflake_developer_service_conn`: A JSON-formatted string containing the connection details required to authenticate and connect to Snowflake.
+    - `synapse_conn_id`: The connection ID for the Synapse connection.
+    - `dataset_collections`: The dataset collections to query for datasets.
+    - `push_results_to_s3`: A boolean to indicate if the results should be pushed to S3.
+                            When set to `False`, the results will be printed to the logs.
+    - `delete_out_of_date_from_s3`: A boolean to indicate if the old files should be 
+                            deleted from S3. When set to `False`, the old files will not 
+                            be deleted, but a message will be logged to indicate that
+                            the files will be deleted when set to `True`.
+    - `push_links_to_synapse`: A boolean to indicate if the links should be pushed to
+                            Synapse. When set to `False`, the links will not be pushed,
+                            but a message will be logged to indicate that the links will
+                            be pushed when set to `True`.
+    - `delete_out_of_date_from_synapse`: A boolean to indicate if the old files should be
+                            deleted from Synapse. When set to `False`, the old files will
+                            not be deleted, but a message will be logged to indicate that
+                            the files will be deleted when set to `True`.
+    - `dataset_collections_for_cleanup`: The dataset collections to query for datasets
+                            to delete the old files from S3. Also consider the Synapse table
+                            where links back to Synapse entities are connected with the S3
+                            object. When this is filled this array is used to filter the
+                            S3 objects which will be considered for deletion.
+    - `aws_conn_id`: The connection ID for the AWS connection. Used to authenticate with S3.
+    """
 
     @task
     def create_root_span(**context) -> Dict:
@@ -242,36 +820,42 @@ def query_synapse_dataset_collection_for_datasets(
         with otel_tracer.start_span("query_synapse_dataset_collection_for_datasets", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
             span.set_attribute("airflow.dataset_collection",
                                dataset_collection)
-            dataset_collection_without_syn = dataset_collection.replace(
-                "syn", "")
-            query = f"""
-            SELECT 
-                REPLACE(ITEM.value:entityId::STRING, 'syn', '') AS DATASET_IDS
-            FROM 
-                SYNAPSE_DATA_WAREHOUSE.SYNAPSE.NODE_LATEST,
-            LATERAL FLATTEN(input => ITEMS) ITEM
-            WHERE
-                node_type = 'datasetcollection'
-                AND id = {dataset_collection_without_syn}
-            """
-            snow_hook = SnowflakeHook(context["params"]["snowflake_developer_service_conn"])
-            ctx = snow_hook.get_conn()
-            cs = ctx.cursor()
 
             otel_logger.info(
                 f"Performing query for dataset_collection {dataset_collection}")
 
+            syn_client = create_syn_client()
+
             try:
-                cs.execute(query)
-                datasets = cs.fetch_pandas_all()
-                cs.close()
                 results = []
-                for syn_id in datasets["DATASET_IDS"].tolist():
-                    results.append({"dataset_id": syn_id.replace("syn", "")})
-                results
+                dataset_collection_entity = syn_client.get(
+                    dataset_collection, downloadFile=False)
+
+                if not hasattr(dataset_collection_entity, "datasetItems") or not dataset_collection_entity.get("datasetItems"):
+                    otel_logger.warning(
+                        f"No datasets found for dataset collection {dataset_collection}")
+                    return results
+                dataset_items = dataset_collection_entity.get("datasetItems")
+                otel_logger.info(dataset_items)
+                for dataset_item in dataset_items:
+                    syn_id = dataset_item["entityId"]
+                    version_number = dataset_item["versionNumber"]
+
+                    try:
+                        dataset = syn_client.get(
+                            f"{syn_id}.{version_number}", downloadFile=False)
+                        span.set_attribute(
+                            "airflow.dataset_name", dataset.name)
+                    except Exception as ex:
+                        otel_logger.warning(
+                            f"Failed to get dataset {syn_id}.{version_number} from Synapse: {ex}")
+
+                    results.append({"dataset_id": syn_id, "dataset_version": version_number,
+                                   "dataset_collection": dataset_collection, "dataset_name": dataset.name})
+
             except Exception as ex:
                 otel_logger.exception(
-                    "Failed to query snowflake for datasets.")
+                    "Failed to query synapse for datasets.")
                 otel_tracer.span_processor.force_flush()
                 otel_logger.handlers[0].flush()
                 raise ex
@@ -292,272 +876,317 @@ def combine_dataset_lists(root_carrier_context: Dict, dataset_ids: List[List[Dic
         Returns:
             The combined list of dataset IDs.
         """
-        with otel_tracer.start_as_current_span("combine_dataset_lists", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+        with otel_tracer.start_as_current_span("combine_dataset_lists", context=TraceContextTextMapPropagator().extract(root_carrier_context)):
             return_value = [
                 dataset for datasets in dataset_ids for dataset in datasets]
         otel_tracer.span_processor.force_flush()
         return return_value
 
     @task
-    def query_snowflake_and_push_croissant_file(root_carrier_context: Dict, dataset_id: str, **context) -> None:
+    def delete_non_current_croissant_file_in_s3(
+        root_carrier_context: Dict, combined_dataset_collection_and_datasets: List[Dict[str, str]], **context
+    ) -> None:
+        """
+        Delete the non-current croissant files from S3. 
+        This is used to remove the old files from S3 that are no longer needed. 
+        A "non-current" file is defined as a croissant JSON LD file which is no longer
+        present in any Dataset collection.
+
+        This can occur if the dataset has been removed from the dataset collection, or
+        if there is a new version of the dataset that has been added to the dataset
+        collection.
+
+        Arguments:
+            root_carrier_context: The root carrier context to use for the trace context.
+            combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+                dataset collections, and dataset versions to use to determine which
+                datasets to delete.
+
+        Returns:
+            None
+        """
+        with otel_tracer.start_as_current_span("delete_non_current_files_from_s3", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            s3_hook = S3Hook(
+                aws_conn_id=context["params"]["aws_conn_id"], region_name=REGION_NAME)
+            bucket_objects = s3_hook.list_keys(bucket_name=BUCKET_NAME)
+
+            objects_to_delete = extract_s3_objects_to_delete(
+                bucket_objects=bucket_objects,
+                dataset_collections_to_consider_for_deletion=context[
+                    "params"]["dataset_collections_for_cleanup"],
+                combined_dataset_collection_and_datasets=combined_dataset_collection_and_datasets,
+            )
+
+            if objects_to_delete:
+                delete_out_of_date_from_s3 = context["params"]["delete_out_of_date_from_s3"]
+                if delete_out_of_date_from_s3:
+                    otel_logger.info(
+                        f"Deleting the following objects from S3: {objects_to_delete}")
+                    s3_hook.delete_objects(
+                        bucket=BUCKET_NAME, keys=objects_to_delete)
+                else:
+                    otel_logger.info(
+                        f"Found objects to delete from S3, but not deleting due to `delete_out_of_date_from_s3` param: {objects_to_delete}")
+            else:
+                otel_logger.info(
+                    "No objects to delete from S3. All objects are current.")
+            otel_tracer.span_processor.force_flush()
+            otel_logger.handlers[0].flush()
+            return None
+
+    @task
+    def delete_non_current_croissant_file_in_synapse(
+        root_carrier_context: Dict, combined_dataset_collection_and_datasets: List[Dict[str, str]], **context
+    ) -> None:
+        """
+        Delete the non-current croissant files from the Synapse table. 
+        This is used to remove the old files from Synapse that are no longer needed. 
+        A "non-current" file is defined as a croissant JSON LD file which is no longer
+        present in any Dataset collection.
+
+        This can occur if the dataset has been removed from the dataset collection, or
+        if there is a new version of the dataset that has been added to the dataset
+        collection.
+
+        Arguments:
+            root_carrier_context: The root carrier context to use for the trace context.
+            combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+                dataset collections, and dataset versions to use to determine which
+                datasets to delete.
+
+        Returns:
+            None
+        """
+        # Warning: Using an authenticated Synapse Client during this section of code
+        with otel_tracer.start_as_current_span("delete_non_current_files_from_s3", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            syn_hook = SynapseHook(context["params"]["synapse_conn_id"])
+            syn_client: Synapse = syn_hook.client
+            syn_client._rest_call = MethodType(
+                _rest_call_replacement, syn_client)
+            dataset_collections_to_consider_for_deletion = [f"'{ds}'" for ds in context[
+                "params"]["dataset_collections_for_cleanup"]]
+            query = f"SELECT * FROM {SYNAPSE_TABLE_FOR_CROISSANT_LINKS} WHERE dataset_collection in ({','.join(dataset_collections_to_consider_for_deletion)})"
+            table_results = syn_client.tableQuery(
+                query=query, resultsAs="csv")
+            table_dataframe = table_results.asDataFrame(
+                rowIdAndVersionInIndex=False)
+            os.remove(table_results.filepath)
+            rows_to_delete = extract_synapse_rows_to_delete(
+                synapse_rows=table_dataframe,
+                combined_dataset_collection_and_datasets=combined_dataset_collection_and_datasets,
+            )
+
+            if rows_to_delete:
+                delete_out_of_date_from_synapse = context["params"]["delete_out_of_date_from_synapse"]
+                if delete_out_of_date_from_synapse:
+                    otel_logger.info(
+                        f"Deleting the following rows from Synapse: {rows_to_delete}")
+                    results_from_synapse = syn_client.tableQuery(
+                        query=f"SELECT ROW_ID, ROW_VERSION FROM {SYNAPSE_TABLE_FOR_CROISSANT_LINKS} WHERE ROW_ID in ({','.join(rows_to_delete)})", resultsAs="csv")
+                    syn_client.delete(results_from_synapse)
+                    os.remove(results_from_synapse.filepath)
+                else:
+                    otel_logger.info(
+                        f"Found rows to delete from Synapse, but not deleting due to `delete_out_of_date_from_synapse` param: {rows_to_delete}")
+            else:
+                otel_logger.info(
+                    "No rows to delete from Synapse. All rows are current.")
+            otel_tracer.span_processor.force_flush()
+            otel_logger.handlers[0].flush()
+            return None
+
+    @task
+    def query_and_push_croissant_file(root_carrier_context: Dict, dataset_id: str, dataset_version: int, dataset_collection: str, **context) -> None:
         """
         Query the Snowflake database for the dataset in order to perform the Croissant
         transformation and push the results to S3.
 
         Arguments:
-            dataset_id: The ID of the dataset to query in Snowflake.
+            root_carrier_context: The root carrier context to use for the trace context.
+            dataset_id: The ID of the dataset to query.
+            dataset_version: The version of the dataset to query.
+            dataset_collection: The ID of the dataset collection to query.
+            context: The context of the DAG run.
+
+        Returns:
+            None
         """
-        with otel_tracer.start_as_current_span("query_snowflake_and_push_croissant_file", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+        with otel_tracer.start_as_current_span("query_and_push_croissant_file", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            push_to_s3 = context["params"]["push_results_to_s3"]
+            push_to_synapse = context["params"]["push_links_to_synapse"]
+            span.set_attribute("airflow.push_results_to_s3", push_to_s3)
+            span.set_attribute(
+                "airflow.push_links_to_synapse", push_to_synapse)
             if not dataset_id:
                 otel_logger.info("No dataset found")
                 otel_tracer.span_processor.force_flush()
                 otel_logger.handlers[0].flush()
                 return
 
-            span.set_attribute("airflow.dataset_id", f"syn{dataset_id}")
-
-            query = f"""
-            WITH entity_metadata AS (
-                SELECT
-                    items,
-                    name,
-                    ANNOTATIONS,
-                    CREATED_ON,
-                    VERSION_NUMBER,
-                    CONCAT('syn', id) as DATASET_SYNAPSE_ID
-                FROM synapse_data_warehouse.synapse.node_latest
-                WHERE id = {dataset_id}
-            ),
-            ids_of_files_belonging_to_dataset AS (
-                SELECT REPLACE(ITEM.value:entityId::STRING, 'syn', '') AS entity_id, ITEM.value:versionNumber::STRING as version_number
-                FROM entity_metadata,
-                LATERAL FLATTEN(input => entity_metadata.ITEMS) ITEM
-            ),
-            files_belonging_to_dataset AS (
-                SELECT
-                    nl.id,
-                    nl.version_number,
-                    vh.value AS VERSION_HISTORY_ENTRY,
-                    nl.ANNOTATIONS,
-                    nl.is_public,
-                    ids_of_files_belonging_to_dataset.entity_id,
-                    ids_of_files_belonging_to_dataset.version_number,
-                    entity_metadata.DATASET_SYNAPSE_ID
-                FROM synapse_data_warehouse.synapse.node_latest AS nl
-                JOIN ids_of_files_belonging_to_dataset
-                    ON nl.id = ids_of_files_belonging_to_dataset.entity_id
-                JOIN entity_metadata
-                    ON TRUE -- Ensures the `DATASET_SYNAPSE_ID` is included
-                CROSS JOIN LATERAL FLATTEN(input => nl.VERSION_HISTORY) vh
-                WHERE nl.is_public = TRUE
-                AND vh.value:versionNumber::STRING = ids_of_files_belonging_to_dataset.version_number
-            ),
-            distribution_pointing_to_files AS (
-                SELECT
-                    CONCAT('syn', id) as SYNAPSE_ID,
-                    DATASET_SYNAPSE_ID,
-                    OBJECT_CONSTRUCT(
-                        '@type', 'FileObject',
-                        '@id', CONCAT('syn', id),
-                        'name', CONCAT('syn', id),
-                        'description', CONCAT(
-                            'Data file associated with ', CONCAT('syn', id)),
-                        'contentUrl', CONCAT(
-                            'https://www.synapse.org/Synapse:', CONCAT('syn', id)),
-                        'encodingFormat', 'application/json',
-                        'md5', COALESCE(VERSION_HISTORY_ENTRY:contentMd5, 'unknown_md5'),
-                        'sha256', 'unknown'
-                    ) AS file_object
-                FROM files_belonging_to_dataset
-            ),
-            -- TODO: Annotation selection must be version dependent, however, the different versions of the annotations are not available in a non-deduplicated table
-            distinct_annotation_keys AS (
-                SELECT DATASET_SYNAPSE_ID, ARRAY_AGG(annotation_key) as annotation_keys FROM (
-                    SELECT
-                        DATASET_SYNAPSE_ID,
-                        key as annotation_key
-                    FROM files_belonging_to_dataset,
-                    LATERAL FLATTEN(input => ANNOTATIONS:annotations) AS annotation
-                    GROUP BY DATASET_SYNAPSE_ID, key
-                ) GROUP BY DATASET_SYNAPSE_ID
-            ),
-            fields_pointing_to_metadata AS (
-                SELECT
-                    DATASET_SYNAPSE_ID,
-                    OBJECT_CONSTRUCT(
-                        '@type', 'RecordSet',
-                        '@id', 'default',
-                        'name', 'default',
-                        'description', 'Metadata for the dataset',
-                        'field', ARRAY_AGG(
-                            OBJECT_CONSTRUCT(
-                                '@type', 'cr:Field',
-                                '@id', CONCAT('metadata', '/', value),
-                                'name', value,
-                                -- 'description', '',
-                                'dataType', 'sc:Text',
-                                'source', OBJECT_CONSTRUCT(
-                                    'fileObject', OBJECT_CONSTRUCT(
-                                        '@id', 'metadata'),
-                                    'extract', OBJECT_CONSTRUCT('column', value)
-                                )
-                            )
-                        )
-                    ) AS record_set
-                FROM distinct_annotation_keys,
-                LATERAL FLATTEN(input => distinct_annotation_keys.annotation_keys)
-                GROUP BY DATASET_SYNAPSE_ID
-            ),
-            croissant_metadata AS (
-                SELECT
-                    entity_metadata.DATASET_SYNAPSE_ID,
-                    entity_metadata.name,
-                    OBJECT_CONSTRUCT(
-                        '@context', OBJECT_CONSTRUCT(
-                            '@language', 'en',
-                            '@vocab', 'https://schema.org/',
-                            'citeAs', 'cr:citeAs',
-                            'column', 'cr:column',
-                            'conformsTo', 'dct:conformsTo',
-                            'cr', 'http://mlcommons.org/croissant/',
-                            'rai', 'http://mlcommons.org/croissant/RAI/',
-                            'data', OBJECT_CONSTRUCT(
-                                '@id', 'cr:data', '@type', '@json'),
-                            'dataType', OBJECT_CONSTRUCT(
-                                '@id', 'cr:dataType', '@type', '@vocab'),
-                            'dct', 'http://purl.org/dc/terms/',
-                            'examples', OBJECT_CONSTRUCT(
-                                '@id', 'cr:examples', '@type', '@json'),
-                            'extract', 'cr:extract',
-                            'field', 'cr:field',
-                            'fileProperty', 'cr:fileProperty',
-                            'fileObject', 'cr:fileObject',
-                            'fileSet', 'cr:fileSet',
-                            'format', 'cr:format',
-                            'includes', 'cr:includes',
-                            'isLiveDataset', 'cr:isLiveDataset',
-                            'jsonPath', 'cr:jsonPath',
-                            'key', 'cr:key',
-                            'md5', 'cr:md5',
-                            'parentField', 'cr:parentField',
-                            'path', 'cr:path',
-                            'recordSet', 'cr:recordSet',
-                            'references', 'cr:references',
-                            'regex', 'cr:regex',
-                            'repeated', 'cr:repeated',
-                            'replace', 'cr:replace',
-                            'sc', 'https://schema.org/',
-                            'separator', 'cr:separator',
-                            'source', 'cr:source',
-                            'subField', 'cr:subField',
-                            'transform', 'cr:transform'
-                        ),
-                        '@type', 'Dataset',
-                        'name', COALESCE(entity_metadata.name, 'Synapse Dataset'),
-                        'description', COALESCE(ANNOTATIONS:annotations:description:value[0], 'A dataset containing annotations for genomic data from Synapse.'),
-                        'url', CONCAT('https://www.synapse.org/Synapse:',
-                                    entity_metadata.DATASET_SYNAPSE_ID),
-                        'citation', COALESCE(ANNOTATIONS:annotations:citation:value[0], 'unknown_citation'),
-                        'datePublished', COALESCE(CREATED_ON, 'unknown_date'),
-                        'license', COALESCE(ANNOTATIONS:annotations:license:value[0], 'unknown_license'),
-                        'version', COALESCE(VERSION_NUMBER, 'unknown_version'),
-                        'distribution', ARRAY_PREPEND(ARRAY_AGG(distribution_pointing_to_files.file_object),
-                                            OBJECT_CONSTRUCT(
-                                                '@type', 'FileObject',
-                                                '@id', 'metadata',
-                                                'name', 'metadata',
-                                                'description', CONCAT(
-                                                    'Metadata associated with ', entity_metadata.DATASET_SYNAPSE_ID),
-                                                'contentUrl', CONCAT(
-                                                    'https://www.synapse.org/Synapse:', entity_metadata.DATASET_SYNAPSE_ID),
-                                                'encodingFormat', 'application/csv',
-                                                'md5', '',
-                                                'sha256', 'unknown'
-                                            )
-                                        ),
-                        'recordSet', fields_pointing_to_metadata.record_set
-                    ) AS metadata
-                FROM
-                    entity_metadata, fields_pointing_to_metadata
-                JOIN
-                    distribution_pointing_to_files
-                ON
-                    distribution_pointing_to_files.DATASET_SYNAPSE_ID = fields_pointing_to_metadata.DATASET_SYNAPSE_ID
-                GROUP BY
-                    entity_metadata.DATASET_SYNAPSE_ID,
-                    entity_metadata.name,
-                    fields_pointing_to_metadata.DATASET_SYNAPSE_ID,
-                    fields_pointing_to_metadata.record_set,
-                    entity_metadata.name,
-                    entity_metadata.ANNOTATIONS,
-                    entity_metadata.CREATED_ON,
-                    entity_metadata.VERSION_NUMBER
-            )
-
-            SELECT
-                metadata::STRING AS croissant_metadata,
-                DATASET_SYNAPSE_ID,
-                name
-            FROM
-                croissant_metadata;
-            """
-            snow_hook = SnowflakeHook(context["params"]["snowflake_developer_service_conn"])
-            ctx = snow_hook.get_conn()
-            cs = ctx.cursor()
-
-            otel_logger.info(
-                f"Performing query for dataset {dataset_id}")
+            span.set_attribute("airflow.dataset_id", f"{dataset_id}")
+            path_to_remove = None
+            syn_client = create_syn_client()
+            file_ids_and_versions_attached_to_dataset = []
 
             try:
-                push_to_s3 = context["params"]["push_results_to_s3"]
-                span.set_attribute("airflow.push_results_to_s3", push_to_s3)
-
-                cs.execute(query)
-                croissant_json_ld_for_dataset = cs.fetch_pandas_all()
-                cs.close()
-
-                if croissant_json_ld_for_dataset.empty:
+                dataset = syn_client.get(
+                    f"{dataset_id}.{dataset_version}", downloadFile=False)
+                span.set_attribute("airflow.dataset_name", dataset.name)
+
+                # The dataset entity in synapse returned a list of files that are
+                # attached to the dataset. This is used to get the file IDs and
+                # versions of the files that are attached to the dataset.
+                if not hasattr(dataset, "datasetItems") or not dataset.get("datasetItems"):
                     span.set_attribute("airflow.croissant_result", False)
-                    otel_logger.info(
-                        f"No content found for {dataset_id}")
-                    otel_tracer.span_processor.force_flush()
-                    otel_logger.handlers[0].flush()
+                    otel_logger.warning(
+                        f"No files found for dataset {dataset_id}.{dataset_version}")
                     return
 
-                synapse_id = croissant_json_ld_for_dataset["DATASET_SYNAPSE_ID"][0]
-                name = croissant_json_ld_for_dataset["NAME"][0]
-
-                span.set_attribute("airflow.dataset_name", name)
-                span.set_attribute("airflow.croissant_result", True)
-
-                if not push_to_s3:
-                    otel_logger.info(
-                        f"Croissant file for [dataset: {name}, id: {synapse_id}]:\n{croissant_json_ld_for_dataset['CROISSANT_METADATA'][0]}")
-                else:
-                    otel_logger.info(
-                        f"Uploading croissant file for [dataset: {name}, id: {synapse_id}]")
-
-                    croissant_metadata_bytes = croissant_json_ld_for_dataset["CROISSANT_METADATA"][0].encode(
-                        'utf-8')
-                    metadata_file = BytesIO(croissant_metadata_bytes)
-                    s3_hook = S3Hook(
-                        aws_conn_id=context["params"]["aws_conn_id"], region_name=REGION_NAME, extra_args={
-                            "ContentType": "application/ld+json"
-                        }
-                    )
-
-                    s3_hook.load_file_obj(file_obj=metadata_file,
-                                          key=f"{name}_{synapse_id}_croissant.jsonld",
-                                          bucket_name=BUCKET_NAME,
-                                          replace=True,
-                                          )
+                dataset_items = dataset.get("datasetItems")
+                for dataset_item in dataset_items:
+                    syn_id = dataset_item["entityId"]
+                    version_number = dataset_item["versionNumber"]
+                    file_ids_and_versions_attached_to_dataset.append(
+                        {"file_id": syn_id, "file_version": version_number})
 
             except Exception as ex:
                 otel_logger.exception(
-                    "Failed to query snowflake and push croissant file to S3.")
+                    f"Failed to query synapse dataset for files. {dataset_id}")
                 otel_tracer.span_processor.force_flush()
                 otel_logger.handlers[0].flush()
                 raise ex
+            finally:
+                if path_to_remove:
+                    os.remove(path_to_remove)
+
+            # Sort so we have a consistent order for the data shown in the croissant file
+            file_ids_and_versions_attached_to_dataset.sort(
+                key=lambda x: x["file_id"])
+
+            otel_logger.info(file_ids_and_versions_attached_to_dataset)
+            files_attached_to_dataset: List[File] = get_file_instances(
+                synapse_files=file_ids_and_versions_attached_to_dataset, syn_client=syn_client)
+
+            if not files_attached_to_dataset:
+                otel_logger.warning(
+                    f"No files found for dataset {dataset_id}.{dataset_version}")
+                span.set_attribute("airflow.croissant_result", False)
+                return
+
+            span.set_attribute("airflow.croissant_result", True)
+
+            # The distribution section of the croissant file is used to describe the
+            # files that are attached to the dataset. The first `metadata` file is
+            # used to describe the metadata associated with the dataset and where to find
+            # it on the Synapse server (The Dataset view)
+            distribution_files = [{
+                "@type": "FileObject",
+                "@id": "metadata",
+                "contentUrl": f"https://www.synapse.org/Synapse:{dataset_id}.{dataset_version}",
+                "name": "metadata",
+                "description": f"Metadata associated with {dataset.name}",
+                "encodingFormat": "application/csv",
+                "md5": "unknown",
+                "sha256": "unknown"
+            }] + construct_distribution_section_for_files(files_attached_to_dataset, **context)
+
+            record_set = construct_record_set_section_for_files(
+                files_attached_to_dataset=files_attached_to_dataset)
+
+            croissant_file = {
+                "@context": {
+                    "@language": "en",
+                    "@vocab": "https://schema.org/",
+                    "citeAs": "cr:citeAs",
+                    "column": "cr:column",
+                    "conformsTo": "dct:conformsTo",
+                    "cr": "http://mlcommons.org/croissant/",
+                    "rai": "http://mlcommons.org/croissant/RAI/",
+                    "data": {
+                        "@id": "cr:data",
+                        "@type": "@json"
+                    },
+                    "dataType": {
+                        "@id": "cr:dataType",
+                        "@type": "@vocab"
+                    },
+                    "dct": "http://purl.org/dc/terms/",
+                    "examples": {
+                        "@id": "cr:examples",
+                        "@type": "@json"
+                    },
+                    "extract": "cr:extract",
+                    "field": "cr:field",
+                    "fileProperty": "cr:fileProperty",
+                    "fileObject": "cr:fileObject",
+                    "fileSet": "cr:fileSet",
+                    "format": "cr:format",
+                    "includes": "cr:includes",
+                    "isLiveDataset": "cr:isLiveDataset",
+                    "jsonPath": "cr:jsonPath",
+                    "key": "cr:key",
+                    "md5": "cr:md5",
+                    "parentField": "cr:parentField",
+                    "path": "cr:path",
+                    "recordSet": "cr:recordSet",
+                    "references": "cr:references",
+                    "regex": "cr:regex",
+                    "repeated": "cr:repeated",
+                    "replace": "cr:replace",
+                    "sc": "https://schema.org/",
+                    "separator": "cr:separator",
+                    "source": "cr:source",
+                    "subField": "cr:subField",
+                    "transform": "cr:transform"
+                },
+                "@type": "Dataset",
+                "@id": f"{dataset_id}.{dataset_version}",
+                "name": f"{dataset.name}",
+                "description": dataset.description if hasattr(dataset, "description") and dataset.description else f"Dataset for {dataset.name}",
+                "url": f"https://www.synapse.org/Synapse:{dataset_id}.{dataset_version}",
+                "datePublished": dataset.modifiedOn,
+                "license": dataset.license[0] if hasattr(dataset, "license") else "unknown_license",
+                "version": dataset_version,
+                "dct:conformsTo": "http://mlcommons.org/croissant/1.0",
+                "distribution": distribution_files,
+                "recordSet": record_set,
+                # https://github.com/nf-osi/nf-metadata-dictionary/blob/main/registered-json-schemas/PortalDataset.json
+                # These fields are derived from: https://raw.githubusercontent.com/nf-osi/nf-metadata-dictionary/refs/heads/main/registered-json-schemas/PortalDataset.json
+                # If a more specific schema is needed, then we can add a mapping for the dataset to point to the schema and pull these in dynamically.
+                "accessType": dataset.accessType[0] if hasattr(dataset, "accessType") else None,
+                "alternateName": dataset.alternateName[0] if hasattr(dataset, "alternateName") else None,
+                "citation": dataset.citation[0] if hasattr(dataset, "citation") else None,
+                "conditionsOfAccess": dataset.conditionsOfAccess[0] if hasattr(dataset, "conditionsOfAccess") else None,
+                "countryOfOrigin": dataset.countryOfOrigin if hasattr(dataset, "countryOfOrigin") else None,
+                "dataType": dataset.dataType if hasattr(dataset, "dataType") else None,
+                "dataUseModifiers": dataset.dataUseModifiers if hasattr(dataset, "dataUseModifiers") else None,
+                "diseaseFocus": dataset.diseaseFocus if hasattr(dataset, "diseaseFocus") else None,
+                "funder": dataset.funder if hasattr(dataset, "funder") else None,
+                "individualCount": dataset.individualCount[0] if hasattr(dataset, "individualCount") else None,
+                "keywords": dataset.keywords if hasattr(dataset, "keywords") else None,
+                "manifestation": dataset.manifestation if hasattr(dataset, "manifestation") else None,
+                "measurementTechnique": dataset.measurementTechnique if hasattr(dataset, "measurementTechnique") else None,
+                "series": dataset.series[0] if hasattr(dataset, "series") else None,
+                "species": dataset.species if hasattr(dataset, "species") else None,
+                "specimenCount": dataset.specimenCount[0] if hasattr(dataset, "specimenCount") else None,
+                "studyId": dataset.studyId[0] if hasattr(dataset, "studyId") else None,
+                "subject": dataset.subject if hasattr(dataset, "subject") else None,
+                "title": dataset.title[0] if hasattr(dataset, "title") else None,
+                "visualizeDataOn": dataset.visualizeDataOn if hasattr(dataset, "visualizeDataOn") else None,
+                "yearProcessed": dataset.yearProcessed[0] if hasattr(dataset, "yearProcessed") else None
+            }
+            delete_none_keys(croissant_file)
+
+            # The logic to push the file to S3 and push to Synapse is occurring within
+            # the same DAG task is to prevent any issues with the large amount of
+            # content that would need to be passed between tasks.
+            s3_key = f"{dataset.name}_{dataset_id}.{dataset_version}_datasetCollection_{dataset_collection}_croissant.jsonld"
+            execute_push_to_s3(dataset=dataset, dataset_id=dataset_id, dataset_version=dataset_version,
+                               s3_key=s3_key, croissant_file=croissant_file, push_to_s3=push_to_s3, **context)
+
+            s3_url = f"https://{BUCKET_NAME}.s3.us-east-1.amazonaws.com/{quote_plus(s3_key)}"
+            execute_push_to_synapse(push_to_synapse=push_to_synapse, dataset=dataset, dataset_id=dataset_id,
+                                    dataset_version=dataset_version, dataset_collection=dataset_collection, s3_url=s3_url, **context)
 
         otel_tracer.span_processor.force_flush()
         otel_logger.handlers[0].flush()
@@ -567,8 +1196,17 @@ def query_snowflake_and_push_croissant_file(root_carrier_context: Dict, dataset_
     datasets = query_synapse_dataset_collection_for_datasets.partial(root_carrier_context=root_carrier_context).expand_kwargs(
         get_dataset_collections(root_carrier_context=root_carrier_context))
 
-    query_snowflake_and_push_croissant_file.partial(root_carrier_context=root_carrier_context).expand_kwargs(
-        combine_dataset_lists(root_carrier_context=root_carrier_context, dataset_ids=datasets))
+    combined_dataset_lists = combine_dataset_lists(
+        root_carrier_context=root_carrier_context, dataset_ids=datasets)
+
+    query_and_push_croissant_file.partial(root_carrier_context=root_carrier_context).expand_kwargs(
+        combined_dataset_lists)
+
+    delete_non_current_croissant_file_in_s3(root_carrier_context=root_carrier_context,
+                                            combined_dataset_collection_and_datasets=combined_dataset_lists)
+
+    delete_non_current_croissant_file_in_synapse(root_carrier_context=root_carrier_context,
+                                                 combined_dataset_collection_and_datasets=combined_dataset_lists)
 
 
 dataset_to_croissant()
diff --git a/requirements-airflow.txt b/requirements-airflow.txt
index e42daf77..92b2091d 100644
--- a/requirements-airflow.txt
+++ b/requirements-airflow.txt
@@ -2,7 +2,7 @@ apache-airflow-providers-amazon ~=9.2.0
 apache-airflow-providers-celery >=3.10.0
 apache-airflow-providers-snowflake ~=6.0.0
 snowflake-connector-python ~=3.13.2
-py-orca[all] == 1.5.0
+py-orca[all] == 1.5.1
 pandas <3.0.0
 slack-sdk >=3.27
 pendulum~=3.0.0
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 679e2dad..2ad75b41 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -1,6 +1,5 @@
-opentelemetry-exporter-otlp ==1.21.0 # make the synapseclient and apache-airflow play nice
-synapseclient ~=4.1.1
-fs-synapse ~=2.0
+synapseclient >=4.7,<5.0
+fs-synapse >=2.0,<3.0
 s3fs ~=2023.5
 metaflow ~=2.9
 boto3 >=1.7.0,<2.0


Reviews:
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T23:57:08Z
user: login: thomasyu888
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T16:57:44Z
user: login: thomasyu888
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T16:58:56Z
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T17:01:16Z
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T00:22:27Z
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T00:22:55Z
user: login: thomasyu888
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T04:24:27Z
user: login: thomasyu888
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T04:30:05Z
user: login: thomasyu888
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T04:40:33Z
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T15:36:47Z
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T15:37:09Z
user: login: BWMac
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T18:26:10Z
user: login: BWMac
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T18:28:19Z
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T18:36:21Z
user: login: thomasyu888
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T18:40:06Z
user: login: thomasyu888
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T18:41:23Z
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T19:46:31Z
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T19:46:42Z
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T19:48:00Z
user: login: SageGJ
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-07T20:21:53Z
user: login: linglp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T23:04:45Z
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T23:16:49Z
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T23:17:37Z
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T23:25:37Z
user: login: linglp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T23:33:53Z
user: login: linglp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T23:42:54Z
user: login: linglp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T23:43:57Z
user: login: BryanFauble
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T23:45:36Z
user: login: linglp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T00:59:30Z
user: login: linglp
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T01:09:40Z
user: login: thomasyu888
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T01:12:18Z

Review comments:
diff_hunk: @@ -165,14 +178,248 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            # TODO: Handle for cases where a file may be included twice within the dataset
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if files_to_find_md5_in_snowflake:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+        ids_of_files = list(files_to_find_md5_in_snowflake.keys())
+        id_and_version_pairs = []
+        for file_id, version in files_to_find_md5_in_snowflake.items():
+            id_and_version_pairs.append(file_id)
+            id_and_version_pairs.append(version)
path: dags/synapse-dataset-to-croissant.py
user: login: BryanFauble
body: TODO: This code will not work if a Dataset contains 2 versions of the same file (Synapse id). Some logical changes are required here
created_at: 2025-04-02T23:57:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -567,8 +778,14 @@ def query_snowflake_and_push_croissant_file(root_carrier_context: Dict, dataset_
     datasets = query_synapse_dataset_collection_for_datasets.partial(root_carrier_context=root_carrier_context).expand_kwargs(
         get_dataset_collections(root_carrier_context=root_carrier_context))
 
+    combined_dataset_lists = combine_dataset_lists(
path: dags/synapse-dataset-to-croissant.py
user: login: thomasyu888
body: Should we just upload a synapse table with the mapping?

Table name: croissant-files
schema:
dataset_id, croissant_link?

That does complicate things a bit, because then you'd need authentication, we can have a separate ticket to do this.
created_at: 2025-04-03T16:57:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -308,236 +613,142 @@ def query_snowflake_and_push_croissant_file(root_carrier_context: Dict, dataset_
             dataset_id: The ID of the dataset to query in Snowflake.
         """
         with otel_tracer.start_as_current_span("query_snowflake_and_push_croissant_file", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            push_to_s3 = context["params"]["push_results_to_s3"]
+            span.set_attribute("airflow.push_results_to_s3", push_to_s3)
             if not dataset_id:
                 otel_logger.info("No dataset found")
                 otel_tracer.span_processor.force_flush()
                 otel_logger.handlers[0].flush()
                 return
 
-            span.set_attribute("airflow.dataset_id", f"syn{dataset_id}")
-
-            query = f"""
-            WITH entity_metadata AS (
-                SELECT
-                    items,
-                    name,
-                    ANNOTATIONS,
-                    CREATED_ON,
-                    VERSION_NUMBER,
-                    CONCAT('syn', id) as DATASET_SYNAPSE_ID
-                FROM synapse_data_warehouse.synapse.node_latest
-                WHERE id = {dataset_id}
-            ),
-            ids_of_files_belonging_to_dataset AS (
-                SELECT REPLACE(ITEM.value:entityId::STRING, 'syn', '') AS entity_id, ITEM.value:versionNumber::STRING as version_number
-                FROM entity_metadata,
-                LATERAL FLATTEN(input => entity_metadata.ITEMS) ITEM
-            ),
-            files_belonging_to_dataset AS (
-                SELECT
-                    nl.id,
-                    nl.version_number,
-                    vh.value AS VERSION_HISTORY_ENTRY,
-                    nl.ANNOTATIONS,
-                    nl.is_public,
-                    ids_of_files_belonging_to_dataset.entity_id,
-                    ids_of_files_belonging_to_dataset.version_number,
-                    entity_metadata.DATASET_SYNAPSE_ID
-                FROM synapse_data_warehouse.synapse.node_latest AS nl
-                JOIN ids_of_files_belonging_to_dataset
-                    ON nl.id = ids_of_files_belonging_to_dataset.entity_id
-                JOIN entity_metadata
-                    ON TRUE -- Ensures the `DATASET_SYNAPSE_ID` is included
-                CROSS JOIN LATERAL FLATTEN(input => nl.VERSION_HISTORY) vh
-                WHERE nl.is_public = TRUE
-                AND vh.value:versionNumber::STRING = ids_of_files_belonging_to_dataset.version_number
-            ),
-            distribution_pointing_to_files AS (
-                SELECT
-                    CONCAT('syn', id) as SYNAPSE_ID,
-                    DATASET_SYNAPSE_ID,
-                    OBJECT_CONSTRUCT(
-                        '@type', 'FileObject',
-                        '@id', CONCAT('syn', id),
-                        'name', CONCAT('syn', id),
-                        'description', CONCAT(
-                            'Data file associated with ', CONCAT('syn', id)),
-                        'contentUrl', CONCAT(
-                            'https://www.synapse.org/Synapse:', CONCAT('syn', id)),
-                        'encodingFormat', 'application/json',
-                        'md5', COALESCE(VERSION_HISTORY_ENTRY:contentMd5, 'unknown_md5'),
-                        'sha256', 'unknown'
-                    ) AS file_object
-                FROM files_belonging_to_dataset
-            ),
-            -- TODO: Annotation selection must be version dependent, however, the different versions of the annotations are not available in a non-deduplicated table
-            distinct_annotation_keys AS (
-                SELECT DATASET_SYNAPSE_ID, ARRAY_AGG(annotation_key) as annotation_keys FROM (
-                    SELECT
-                        DATASET_SYNAPSE_ID,
-                        key as annotation_key
-                    FROM files_belonging_to_dataset,
-                    LATERAL FLATTEN(input => ANNOTATIONS:annotations) AS annotation
-                    GROUP BY DATASET_SYNAPSE_ID, key
-                ) GROUP BY DATASET_SYNAPSE_ID
-            ),
-            fields_pointing_to_metadata AS (
-                SELECT
-                    DATASET_SYNAPSE_ID,
-                    OBJECT_CONSTRUCT(
-                        '@type', 'RecordSet',
-                        '@id', 'default',
-                        'name', 'default',
-                        'description', 'Metadata for the dataset',
-                        'field', ARRAY_AGG(
-                            OBJECT_CONSTRUCT(
-                                '@type', 'cr:Field',
-                                '@id', CONCAT('metadata', '/', value),
-                                'name', value,
-                                -- 'description', '',
-                                'dataType', 'sc:Text',
-                                'source', OBJECT_CONSTRUCT(
-                                    'fileObject', OBJECT_CONSTRUCT(
-                                        '@id', 'metadata'),
-                                    'extract', OBJECT_CONSTRUCT('column', value)
-                                )
-                            )
-                        )
-                    ) AS record_set
-                FROM distinct_annotation_keys,
-                LATERAL FLATTEN(input => distinct_annotation_keys.annotation_keys)
-                GROUP BY DATASET_SYNAPSE_ID
-            ),
-            croissant_metadata AS (
-                SELECT
-                    entity_metadata.DATASET_SYNAPSE_ID,
-                    entity_metadata.name,
-                    OBJECT_CONSTRUCT(
-                        '@context', OBJECT_CONSTRUCT(
-                            '@language', 'en',
-                            '@vocab', 'https://schema.org/',
-                            'citeAs', 'cr:citeAs',
-                            'column', 'cr:column',
-                            'conformsTo', 'dct:conformsTo',
-                            'cr', 'http://mlcommons.org/croissant/',
-                            'rai', 'http://mlcommons.org/croissant/RAI/',
-                            'data', OBJECT_CONSTRUCT(
-                                '@id', 'cr:data', '@type', '@json'),
-                            'dataType', OBJECT_CONSTRUCT(
-                                '@id', 'cr:dataType', '@type', '@vocab'),
-                            'dct', 'http://purl.org/dc/terms/',
-                            'examples', OBJECT_CONSTRUCT(
-                                '@id', 'cr:examples', '@type', '@json'),
-                            'extract', 'cr:extract',
-                            'field', 'cr:field',
-                            'fileProperty', 'cr:fileProperty',
-                            'fileObject', 'cr:fileObject',
-                            'fileSet', 'cr:fileSet',
-                            'format', 'cr:format',
-                            'includes', 'cr:includes',
-                            'isLiveDataset', 'cr:isLiveDataset',
-                            'jsonPath', 'cr:jsonPath',
-                            'key', 'cr:key',
-                            'md5', 'cr:md5',
-                            'parentField', 'cr:parentField',
-                            'path', 'cr:path',
-                            'recordSet', 'cr:recordSet',
-                            'references', 'cr:references',
-                            'regex', 'cr:regex',
-                            'repeated', 'cr:repeated',
-                            'replace', 'cr:replace',
-                            'sc', 'https://schema.org/',
-                            'separator', 'cr:separator',
-                            'source', 'cr:source',
-                            'subField', 'cr:subField',
-                            'transform', 'cr:transform'
-                        ),
-                        '@type', 'Dataset',
-                        'name', COALESCE(entity_metadata.name, 'Synapse Dataset'),
-                        'description', COALESCE(ANNOTATIONS:annotations:description:value[0], 'A dataset containing annotations for genomic data from Synapse.'),
-                        'url', CONCAT('https://www.synapse.org/Synapse:',
-                                    entity_metadata.DATASET_SYNAPSE_ID),
-                        'citation', COALESCE(ANNOTATIONS:annotations:citation:value[0], 'unknown_citation'),
-                        'datePublished', COALESCE(CREATED_ON, 'unknown_date'),
-                        'license', COALESCE(ANNOTATIONS:annotations:license:value[0], 'unknown_license'),
-                        'version', COALESCE(VERSION_NUMBER, 'unknown_version'),
-                        'distribution', ARRAY_PREPEND(ARRAY_AGG(distribution_pointing_to_files.file_object),
-                                            OBJECT_CONSTRUCT(
-                                                '@type', 'FileObject',
-                                                '@id', 'metadata',
-                                                'name', 'metadata',
-                                                'description', CONCAT(
-                                                    'Metadata associated with ', entity_metadata.DATASET_SYNAPSE_ID),
-                                                'contentUrl', CONCAT(
-                                                    'https://www.synapse.org/Synapse:', entity_metadata.DATASET_SYNAPSE_ID),
-                                                'encodingFormat', 'application/csv',
-                                                'md5', '',
-                                                'sha256', 'unknown'
-                                            )
-                                        ),
-                        'recordSet', fields_pointing_to_metadata.record_set
-                    ) AS metadata
-                FROM
-                    entity_metadata, fields_pointing_to_metadata
-                JOIN
-                    distribution_pointing_to_files
-                ON
-                    distribution_pointing_to_files.DATASET_SYNAPSE_ID = fields_pointing_to_metadata.DATASET_SYNAPSE_ID
-                GROUP BY
-                    entity_metadata.DATASET_SYNAPSE_ID,
-                    entity_metadata.name,
-                    fields_pointing_to_metadata.DATASET_SYNAPSE_ID,
-                    fields_pointing_to_metadata.record_set,
-                    entity_metadata.name,
-                    entity_metadata.ANNOTATIONS,
-                    entity_metadata.CREATED_ON,
-                    entity_metadata.VERSION_NUMBER
-            )
-
-            SELECT
-                metadata::STRING AS croissant_metadata,
-                DATASET_SYNAPSE_ID,
-                name
-            FROM
-                croissant_metadata;
-            """
-            snow_hook = SnowflakeHook(context["params"]["snowflake_developer_service_conn"])
-            ctx = snow_hook.get_conn()
-            cs = ctx.cursor()
-
-            otel_logger.info(
-                f"Performing query for dataset {dataset_id}")
+            span.set_attribute("airflow.dataset_id", f"{dataset_id}")
+            path_to_remove = None
+            syn_client = create_syn_client()
+            file_ids_and_versions_attached_to_dataset = []
 
             try:
-                push_to_s3 = context["params"]["push_results_to_s3"]
-                span.set_attribute("airflow.push_results_to_s3", push_to_s3)
+                dataset = syn_client.get(
+                    f"{dataset_id}.{dataset_version}", downloadFile=False)
+                span.set_attribute("airflow.dataset_name", dataset.name)
 
-                cs.execute(query)
-                croissant_json_ld_for_dataset = cs.fetch_pandas_all()
-                cs.close()
-
-                if croissant_json_ld_for_dataset.empty:
+                if not hasattr(dataset, "datasetItems") or not dataset.get("datasetItems"):
                     span.set_attribute("airflow.croissant_result", False)
-                    otel_logger.info(
-                        f"No content found for {dataset_id}")
-                    otel_tracer.span_processor.force_flush()
-                    otel_logger.handlers[0].flush()
+                    otel_logger.warning(
+                        f"No files found for dataset {dataset_id}.{dataset_version}")
                     return
+                dataset_items = dataset.get("datasetItems")
+                for dataset_item in dataset_items:
+                    syn_id = dataset_item["entityId"]
+                    version_number = dataset_item["versionNumber"]
+                    file_ids_and_versions_attached_to_dataset.append(
+                        {"file_id": syn_id, "file_version": version_number})
 
-                synapse_id = croissant_json_ld_for_dataset["DATASET_SYNAPSE_ID"][0]
-                name = croissant_json_ld_for_dataset["NAME"][0]
-
-                span.set_attribute("airflow.dataset_name", name)
-                span.set_attribute("airflow.croissant_result", True)
+            except Exception as ex:
+                otel_logger.exception(
+                    f"Failed to query synapse dataset for files. {dataset_id}")
+                otel_tracer.span_processor.force_flush()
+                otel_logger.handlers[0].flush()
+                raise ex
+            finally:
+                if path_to_remove:
+                    os.remove(path_to_remove)
+
+            file_ids_and_versions_attached_to_dataset.sort(
+                key=lambda x: x["file_id"])
+
+            otel_logger.info(file_ids_and_versions_attached_to_dataset)
+            files_attached_to_dataset: List[File] = get_file_instances(
+                synapse_files=file_ids_and_versions_attached_to_dataset, syn_client=syn_client)
+            if not files_attached_to_dataset:
+                otel_logger.warning(
+                    f"No files found for dataset {dataset_id}.{dataset_version}")
+                span.set_attribute("airflow.croissant_result", False)
+                return
+            span.set_attribute("airflow.croissant_result", True)
+
+            distribution_files = [{
+                "@type": "FileObject",
+                "@id": "metadata",
+                "contentUrl": f"https://www.synapse.org/Synapse:{dataset_id}.{dataset_version}",
+                "name": "metadata",
+                "description": f"Metadata associated with {dataset.name}",
+                "encodingFormat": "application/csv",
+                "md5": "unknown",
+                "sha256": "unknown"
+            }] + construct_distribution_section_for_files(files_attached_to_dataset, **context)
+
+            record_set = construct_record_set_section_for_files(
+                files_attached_to_dataset=files_attached_to_dataset)
+
+            croissant_file = {
+                "@context": {
+                    "@language": "en",
+                    "@vocab": "https://schema.org/",
+                    "citeAs": "cr:citeAs",
+                    "column": "cr:column",
+                    "conformsTo": "dct:conformsTo",
+                    "cr": "http://mlcommons.org/croissant/",
+                    "rai": "http://mlcommons.org/croissant/RAI/",
+                    "data": {
+                        "@id": "cr:data",
+                        "@type": "@json"
+                    },
+                    "dataType": {
+                        "@id": "cr:dataType",
+                        "@type": "@vocab"
+                    },
+                    "dct": "http://purl.org/dc/terms/",
+                    "examples": {
+                        "@id": "cr:examples",
+                        "@type": "@json"
+                    },
+                    "extract": "cr:extract",
+                    "field": "cr:field",
+                    "fileProperty": "cr:fileProperty",
+                    "fileObject": "cr:fileObject",
+                    "fileSet": "cr:fileSet",
+                    "format": "cr:format",
+                    "includes": "cr:includes",
+                    "isLiveDataset": "cr:isLiveDataset",
+                    "jsonPath": "cr:jsonPath",
+                    "key": "cr:key",
+                    "md5": "cr:md5",
+                    "parentField": "cr:parentField",
+                    "path": "cr:path",
+                    "recordSet": "cr:recordSet",
+                    "references": "cr:references",
+                    "regex": "cr:regex",
+                    "repeated": "cr:repeated",
+                    "replace": "cr:replace",
+                    "sc": "https://schema.org/",
+                    "separator": "cr:separator",
+                    "source": "cr:source",
+                    "subField": "cr:subField",
+                    "transform": "cr:transform"
+                },
+                "@type": "Dataset",
+                "@id": f"{dataset_id}.{dataset_version}",
+                "name": f"{dataset.name}",
+                "description": f"Dataset for {dataset.name}",
path: dags/synapse-dataset-to-croissant.py
user: login: thomasyu888
body: Datasets actually also have a "description" property.  it might be helpful to pull that down?
created_at: 2025-04-03T16:58:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -567,8 +778,14 @@ def query_snowflake_and_push_croissant_file(root_carrier_context: Dict, dataset_
     datasets = query_synapse_dataset_collection_for_datasets.partial(root_carrier_context=root_carrier_context).expand_kwargs(
         get_dataset_collections(root_carrier_context=root_carrier_context))
 
+    combined_dataset_lists = combine_dataset_lists(
path: dags/synapse-dataset-to-croissant.py
user: login: BryanFauble
body: Yeah. I was intending to knock this out. Use the existing service account to push out data to a synapse table.

Schema would need to be:
Dataset syn id
Dataset Version
Dataset collection syn id
Croissant file s3 link
created_at: 2025-04-03T17:01:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,552 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
path: dags/synapse-dataset-to-croissant.py
user: login: BryanFauble
body: This portion is the only usage of Snowflake in this script
created_at: 2025-04-04T00:22:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,552 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
+        elif not file_md5s.empty and int(file.id.replace("syn", "")) in file_md5s["ID"].values:
+            file_md5 = file_md5s.loc[file_md5s["ID"] == int(
+                file.id.replace("syn", "")), "CONTENTMD5"].values[0]
+        else:
+            file_md5 = "unknown_md5"
+
+        distribution_files.append(
+            {
+                "@type": "FileObject",
+                "@id": f"{file.id}.{file.version_number}",
+                "name": f"{file.name}",
+                "description": f"Data file associated with {file.name}",
+                "contentUrl": f"https://www.synapse.org/Synapse:{file.id}.{file.version_number}",
+                "encodingFormat": "application/json",
+                "md5": file_md5,
+                "sha256": "unknown",
+            }
+        )
+    return distribution_files
+
+
+def construct_record_set_section_for_files(files_attached_to_dataset: List[File]) -> Dict[str, str]:
+    """
+    Construct the record set section for the files attached to the dataset. This is used
+    to extract the keys from the annotations of the files and create a Field for each
+    key.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+
+    Returns:
+        The record set section for the files attached to the dataset.
+    """
+    unique_annotation_keys = set()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        file_annotations = file.annotations.keys()
+        for key in file_annotations:
+            unique_annotation_keys.add(key)
+
+    metadata_fields = []
+    # Sort the set of unique_annotation_keys ignoring case, but keep the case in the result
+    unique_annotation_keys = sorted(unique_annotation_keys, key=str.casefold)
+
+    for key in unique_annotation_keys:
+        metadata_fields.append(
+            {
+                "@type": "Field",
+                "@id": f"metadata/{key}",
+                "name": key,
+                "description": "",
+                "dataType": "sc:Text",
+                "source": {
+                    "fileObject": {"@id": "metadata"},
+                    "extract": {"column": key}
+                }
+            }
+        )
+
+    return {
+        "@type": "RecordSet",
+        "@id": "default",
+        "name": "default",
+        "description": "Metadata for the dataset",
+        "field": metadata_fields
+    }
+
+
+def extract_s3_objects_to_delete(bucket_objects: List[str], dataset_collections_to_consider_for_deletion: List[str], combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the S3 objects to delete from the S3 bucket. This is done by retrieving
+    all of the objects in the S3 bucket and checking if the object matches one of the
+    datasets that will be kept after the full DAG as been run. Every object in the S3
+    bucket that matches the pattern `_syn####.###_` and
+    `_datasetCollection_syn####_` will be considered for deletion. The DAG param
+    `dataset_collections_for_cleanup` is used to filter the objects that will be
+    considered for deletion as the second filter.
+
+    The last filter enforces that each object in the S3 bucket must match a dataset
+    id, dataset version, dataset collection, and dataset name in the
+    `combined_dataset_collection_and_datasets` list. If the object does not match
+    any of the objects in the list, then it will be marked for deletion.
+
+    Arguments:
+        bucket_objects: The list of objects in the S3 bucket.
+        dataset_collections_to_consider_for_deletion: The dataset collections to
+            consider for deletion. This is passed in from the
+            `dataset_collections_for_cleanup` DAG parameter.
+    """
+    objects_to_delete = []
+
+    if not bucket_objects:
+        otel_logger.info(
+            "No objects found in S3.")
+        return objects_to_delete
+
+    for bucket_object in bucket_objects:
+        dataset_id_and_version = re.search(
+            DATASET_ID_VERSION_PATTERN, bucket_object)
+        dataset_collection = re.search(
+            DATASET_COLLECTION_PATTERN, bucket_object)
+
+        if not dataset_id_and_version or not dataset_collection:
+            otel_logger.info(
+                f"Object {bucket_object} does not match the pattern. Skipping.")
+            continue
+
+        dataset_collection = dataset_collection.group(0).replace(
+            "_datasetCollection_", "").replace("_", "")
+
+        if dataset_collection not in dataset_collections_to_consider_for_deletion:
+            otel_logger.info(
+                f"Object {bucket_object} does not match a dataset collection present in `dataset_collections_for_cleanup`. Skipping.")
+            continue
+
+        dataset_id, dataset_version = dataset_id_and_version.group(0).split(
+            ".")
+        dataset_id = dataset_id.replace("_", "")
+        dataset_version = dataset_version.replace("_", "")
+        dataset_name = bucket_object.split(dataset_id_and_version.group(0))[0]
+
+        match_found_for_object = False
+
+        for combined_dataset_collection_and_dataset in combined_dataset_collection_and_datasets:
+            if (dataset_collection == combined_dataset_collection_and_dataset["dataset_collection"] and
+                    dataset_id == combined_dataset_collection_and_dataset["dataset_id"] and
+                    dataset_version == str(combined_dataset_collection_and_dataset["dataset_version"]) and
+                    dataset_name == combined_dataset_collection_and_dataset["dataset_name"]):
+                match_found_for_object = True
+                break
+        if not match_found_for_object:
+            objects_to_delete.append(bucket_object)
+
+    return objects_to_delete
+
+
+def extract_synapse_rows_to_delete(synapse_rows: DataFrame, combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the rows to delete from the Synapse table. This is done by retrieving the
+    dataset ids along with their versions from the dataset collections passed in via
+    the `dataset_collections_for_cleanup` DAG parameter. The rows to delete are
+    determined by checking if the dataset id and version are present in the
+    `combined_dataset_collection_and_datasets` list. If the dataset id and version
+    are not present in the list, then the row is marked for deletion.
+
+    Arguments:
+        synapse_rows: A Pandas DataFrame with the columns `dataset`, `dataset_version`,
+            `dataset_collection`, and `ROW_ID`.
+        combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+            dataset collections, and dataset versions to use to determine which
+            datasets to delete.
+
+    Returns:
+        The list of rows to delete from the Synapse table.
+    """
+    rows_to_delete = []
+
+    if synapse_rows.empty:
+        otel_logger.info(
+            "No rows found in Synapse.")
+        return rows_to_delete
+
+    for synapse_row in synapse_rows.itertuples():
+        dataset_collection = synapse_row.dataset_collection
+        dataset_id = synapse_row.dataset
+        dataset_version = synapse_row.dataset_version
+        row_id = synapse_row.ROW_ID
+
+        match_found_for_object = False
+
+        for combined_dataset_collection_and_dataset in combined_dataset_collection_and_datasets:
+            if (dataset_collection == combined_dataset_collection_and_dataset["dataset_collection"] and
+                    dataset_id == combined_dataset_collection_and_dataset["dataset_id"] and
+                    dataset_version == combined_dataset_collection_and_dataset["dataset_version"]):
+                match_found_for_object = True
+                break
+        if not match_found_for_object:
+            rows_to_delete.append(row_id)
+
+    return rows_to_delete
+
+
+def execute_push_to_s3(dataset: Entity, dataset_id: str, dataset_version: str, s3_key: str, croissant_file: Dict[str, Any], push_to_s3: bool, **context) -> None:
+    """
+    Handle the push to S3 of the croissant file. This is done by using the S3Hook to
+    upload the file to S3. The S3 bucket is stored in the `org-sagebase-dpe-prod` AWS
+    account.
+
+    Arguments:
+        dataset: The dataset to push to S3.
+        dataset_id: The ID of the dataset.
+        dataset_version: The version of the dataset.
+        s3_key: The S3 key to use to push the file to S3.
+        croissant_file: The croissant file to push to S3.
+        push_to_s3: A boolean to indicate if the results should be pushed to S3.
+            When set to `False`, the results will be printed to the logs.
+        context: The context of the DAG run.
+
+    Returns:
+
+    """
+    try:
+        if not push_to_s3:
+            otel_logger.info(
+                f"Croissant file for [dataset: {dataset.name}, id: {dataset_id}.{dataset_version}]:\n{json.dumps(croissant_file)}")
+            return
+
+        otel_logger.info(
+            f"Uploading croissant file for [dataset: {dataset.name}, id: {dataset_id}.{dataset_version}]")
+
+        croissant_metadata_bytes = json.dumps(croissant_file).encode(
+            'utf-8')
+        metadata_file = BytesIO(croissant_metadata_bytes)
+        s3_hook = S3Hook(
+            aws_conn_id=context["params"]["aws_conn_id"], region_name=REGION_NAME, extra_args={
+                "ContentType": "application/ld+json"
+            }
+        )
+
+        otel_logger.info(
+            f"Uploading croissant file to S3: {s3_key}")
+        s3_hook.load_file_obj(file_obj=metadata_file,
+                              key=s3_key,
+                              bucket_name=BUCKET_NAME,
+                              replace=True,
+                              )
+    except Exception as ex:
+        otel_logger.exception(
+            "Failed to query snowflake and push croissant file to S3.")
+        otel_tracer.span_processor.force_flush()
+        otel_logger.handlers[0].flush()
+        raise ex
+
+
+def execute_push_to_synapse(push_to_synapse: bool, dataset: Entity, dataset_id: str, dataset_version: str, dataset_collection: str, s3_url: str, **context) -> None:
+    """
+    Handle the push to Synapse of the croissant file link. This is done by using
+    an unauthenticated Synapse client to first query the table to determine if an
+    update is needed. If the link already exists with the expected S3 URL, then
+    skip the update. If the link does not exist or the S3 URL is different, then
+    update the link with the new S3 URL using the authenticated Synapse client.
+
+    Arguments:
+        push_to_synapse: A boolean to indicate if the results should be pushed to
+            Synapse. When set to `False`, the results will be printed to the logs.
+        dataset: The dataset to push to Synapse.
+        dataset_id: The ID of the dataset.
+        dataset_version: The version of the dataset.
+        dataset_collection: The ID of the dataset collection where the dataset came from.
+        s3_url: The S3 URL to use for the value of the cell in the table.
+        syn_client: The unauthenticated Synapse client to use to query the table.
+        context: The context of the DAG run.
+
+    Returns:
+        None
+    """
+    try:
+        if not push_to_synapse:
+            otel_logger.info(
+                f"Croissant file link for [dataset: {dataset.name}, id: {dataset_id}.{dataset_version}]: {s3_url}")
+            return
+
+        otel_logger.info(
+            f"Uploading croissant file link to Synapse table {SYNAPSE_TABLE_FOR_CROISSANT_LINKS}"
+        )
+
+        # TODO: When 4.8.0 of the SYNPY client is released this may be
+        # replaced by the upsert functionality
+
+        # Warning: Using an authenticated Synapse Client during this section of code
+        syn_hook = SynapseHook(
+            context["params"]["synapse_conn_id"])
+        authenticated_syn_client: Synapse = syn_hook.client
+        authenticated_syn_client._rest_call = MethodType(
+            _rest_call_replacement, authenticated_syn_client)
+        existing_row = authenticated_syn_client.tableQuery(
+            query=f"SELECT * FROM {SYNAPSE_TABLE_FOR_CROISSANT_LINKS} WHERE dataset = '{dataset_id}' AND dataset_version = {dataset_version} AND dataset_collection = '{dataset_collection}'", resultsAs="csv")
+        existing_row_df = existing_row.asDataFrame()
+        os.remove(existing_row.filepath)
+        if not existing_row_df.empty and existing_row_df["croissant_file_s3_object"].values[0] == s3_url:
+            otel_logger.info(
+                f"Croissant file link already exists in Synapse table {SYNAPSE_TABLE_FOR_CROISSANT_LINKS}. Skipping.")
+            return
+
+        df = DataFrame(
+            data={
+                "dataset": [dataset_id],
+                "dataset_version": [dataset_version],
+                "dataset_collection": [dataset_collection],
+                "croissant_file_s3_object": [s3_url]
+            }
+        )
+        temp_dir = tempfile.mkdtemp()
+        filepath = os.path.join(temp_dir, "table.csv")
+
+        schema = authenticated_syn_client.get(
+            SYNAPSE_TABLE_FOR_CROISSANT_LINKS)
+        if existing_row_df.empty:
+            # If the row does not exist, create a new row
+            authenticated_syn_client.store(
+                Table(schema=schema, values=df, filepath=filepath))
+        else:
+            # Update the existing row with the new value
+            existing_row_df["croissant_file_s3_object"] = [s3_url]
+            authenticated_syn_client.store(
+                Table(schema=schema, values=existing_row_df, filepath=filepath))
+
+        os.remove(filepath)
+    except Exception as ex:
+        otel_logger.exception(
+            "Failed to push croissant file link to Synapse.")
+        otel_tracer.span_processor.force_flush()
+        otel_logger.handlers[0].flush()
+        raise ex
+
+
 @dag(**dag_config)
 def dataset_to_croissant() -> None:
-    """Execute a query on Snowflake and report the results to a Synapse table."""
+    """Execute a query on Snowflake and report the results to a Synapse table.
+
+
+    DAG Parameters:
path: dags/synapse-dataset-to-croissant.py
user: login: BryanFauble
body: By Adding this to this docstring it will show up in the airflow UI now!

![image](https://github.com/user-attachments/assets/0dee4924-bb5c-42c8-b471-894955a19922)

created_at: 2025-04-04T00:22:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,552 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
path: dags/synapse-dataset-to-croissant.py
user: login: thomasyu888
body: I wonder if snowflake has limits to the number of `value, value2` in the query line
created_at: 2025-04-04T04:24:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,552 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
path: dags/synapse-dataset-to-croissant.py
user: login: thomasyu888
body: Is this just the case where we download the file?
created_at: 2025-04-04T04:30:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,552 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
+        elif not file_md5s.empty and int(file.id.replace("syn", "")) in file_md5s["ID"].values:
+            file_md5 = file_md5s.loc[file_md5s["ID"] == int(
+                file.id.replace("syn", "")), "CONTENTMD5"].values[0]
+        else:
+            file_md5 = "unknown_md5"
+
+        distribution_files.append(
+            {
+                "@type": "FileObject",
+                "@id": f"{file.id}.{file.version_number}",
+                "name": f"{file.name}",
+                "description": f"Data file associated with {file.name}",
+                "contentUrl": f"https://www.synapse.org/Synapse:{file.id}.{file.version_number}",
+                "encodingFormat": "application/json",
path: dags/synapse-dataset-to-croissant.py
user: login: thomasyu888
body: Is this encoding format for the associated file...?
created_at: 2025-04-04T04:40:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,552 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
path: dags/synapse-dataset-to-croissant.py
user: login: BryanFauble
body: We are not downloading the file in this case, but we can get the MD5 from synapse in some cases. Those cases are when the file can be anonymously downloaded. In other cases we need to pull the data from Snowflake 
created_at: 2025-04-04T15:36:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,552 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
+        elif not file_md5s.empty and int(file.id.replace("syn", "")) in file_md5s["ID"].values:
+            file_md5 = file_md5s.loc[file_md5s["ID"] == int(
+                file.id.replace("syn", "")), "CONTENTMD5"].values[0]
+        else:
+            file_md5 = "unknown_md5"
+
+        distribution_files.append(
+            {
+                "@type": "FileObject",
+                "@id": f"{file.id}.{file.version_number}",
+                "name": f"{file.name}",
+                "description": f"Data file associated with {file.name}",
+                "contentUrl": f"https://www.synapse.org/Synapse:{file.id}.{file.version_number}",
+                "encodingFormat": "application/json",
path: dags/synapse-dataset-to-croissant.py
user: login: BryanFauble
body: Good point. I added it to the ticket and will look to pull the right format based on the file.
created_at: 2025-04-04T15:37:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,43 +1,56 @@
 """
-This DAG queries snowflake for the given dataset collections to look for datasets.
+This DAG interacts with a few different services to accomplish the following:
 
-For each dataset in the dataset collections, it queries the dataset in Snowflake and 
+- (Synapse, Unauthenticated) Given list of dataset collections query Synapse to retrieve Anonymously accessible datasets
+- (Synapse, Unauthenticated) For each dataset in the dataset collections, query Synapse Anonymously to retrieve metadata about each dataset
+- (Synapse, Unauthenticated) For each Anonymously accessible dataset, query Synapse Anonymously to retrieve the files attached to the dataset
+- (Snowflake, Authenticated) For each file which is not Anonymously downloadable, query Snowflake to retrieve the content_md5 (Authenticated calls to the datawarehouse)
+- (S3, Authenticated) For each dataset push an object a public S3 bucket in the `org-sagebase-dpe-prod` AWS account
+- (Synapse, Authenticated) For each dataset query a Synapse table which contains links to the S3 object
+- (Synapse, Authenticated) Delete rows from the Synapse table which are not in a dataset collection
+- (Synapse, Authenticated) For each dataset push a link for the S3 object to a Synapse table via the authenticated Synapse client
+- (S3, Authenticated) Delete S3 objects from the S3 bucket which are not in a dataset collection
+
+
+To add more datasets to be proceed by this dag:
path: dags/synapse-dataset-to-croissant.py
user: login: BWMac
body: 
```suggestion
To add more datasets to be processed by this dag:
```
created_at: 2025-04-07T18:05:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,552 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
+        elif not file_md5s.empty and int(file.id.replace("syn", "")) in file_md5s["ID"].values:
+            file_md5 = file_md5s.loc[file_md5s["ID"] == int(
+                file.id.replace("syn", "")), "CONTENTMD5"].values[0]
+        else:
+            file_md5 = "unknown_md5"
+
+        distribution_files.append(
+            {
+                "@type": "FileObject",
+                "@id": f"{file.id}.{file.version_number}",
+                "name": f"{file.name}",
+                "description": f"Data file associated with {file.name}",
+                "contentUrl": f"https://www.synapse.org/Synapse:{file.id}.{file.version_number}",
+                "encodingFormat": "application/json",
+                "md5": file_md5,
+                "sha256": "unknown",
+            }
+        )
+    return distribution_files
+
+
+def construct_record_set_section_for_files(files_attached_to_dataset: List[File]) -> Dict[str, str]:
+    """
+    Construct the record set section for the files attached to the dataset. This is used
+    to extract the keys from the annotations of the files and create a Field for each
+    key.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+
+    Returns:
+        The record set section for the files attached to the dataset.
+    """
+    unique_annotation_keys = set()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        file_annotations = file.annotations.keys()
+        for key in file_annotations:
+            unique_annotation_keys.add(key)
+
+    metadata_fields = []
+    # Sort the set of unique_annotation_keys ignoring case, but keep the case in the result
+    unique_annotation_keys = sorted(unique_annotation_keys, key=str.casefold)
+
+    for key in unique_annotation_keys:
+        metadata_fields.append(
+            {
+                "@type": "Field",
+                "@id": f"metadata/{key}",
+                "name": key,
+                "description": "",
+                "dataType": "sc:Text",
+                "source": {
+                    "fileObject": {"@id": "metadata"},
+                    "extract": {"column": key}
+                }
+            }
+        )
+
+    return {
+        "@type": "RecordSet",
+        "@id": "default",
+        "name": "default",
+        "description": "Metadata for the dataset",
+        "field": metadata_fields
+    }
+
+
+def extract_s3_objects_to_delete(bucket_objects: List[str], dataset_collections_to_consider_for_deletion: List[str], combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the S3 objects to delete from the S3 bucket. This is done by retrieving
+    all of the objects in the S3 bucket and checking if the object matches one of the
+    datasets that will be kept after the full DAG as been run. Every object in the S3
+    bucket that matches the pattern `_syn####.###_` and
+    `_datasetCollection_syn####_` will be considered for deletion. The DAG param
+    `dataset_collections_for_cleanup` is used to filter the objects that will be
+    considered for deletion as the second filter.
+
+    The last filter enforces that each object in the S3 bucket must match a dataset
+    id, dataset version, dataset collection, and dataset name in the
+    `combined_dataset_collection_and_datasets` list. If the object does not match
+    any of the objects in the list, then it will be marked for deletion.
+
+    Arguments:
+        bucket_objects: The list of objects in the S3 bucket.
+        dataset_collections_to_consider_for_deletion: The dataset collections to
+            consider for deletion. This is passed in from the
+            `dataset_collections_for_cleanup` DAG parameter.
+    """
+    objects_to_delete = []
+
+    if not bucket_objects:
+        otel_logger.info(
+            "No objects found in S3.")
+        return objects_to_delete
+
+    for bucket_object in bucket_objects:
+        dataset_id_and_version = re.search(
+            DATASET_ID_VERSION_PATTERN, bucket_object)
+        dataset_collection = re.search(
+            DATASET_COLLECTION_PATTERN, bucket_object)
+
+        if not dataset_id_and_version or not dataset_collection:
+            otel_logger.info(
+                f"Object {bucket_object} does not match the pattern. Skipping.")
+            continue
+
+        dataset_collection = dataset_collection.group(0).replace(
+            "_datasetCollection_", "").replace("_", "")
+
+        if dataset_collection not in dataset_collections_to_consider_for_deletion:
+            otel_logger.info(
+                f"Object {bucket_object} does not match a dataset collection present in `dataset_collections_for_cleanup`. Skipping.")
+            continue
+
+        dataset_id, dataset_version = dataset_id_and_version.group(0).split(
+            ".")
+        dataset_id = dataset_id.replace("_", "")
+        dataset_version = dataset_version.replace("_", "")
+        dataset_name = bucket_object.split(dataset_id_and_version.group(0))[0]
+
+        match_found_for_object = False
+
+        for combined_dataset_collection_and_dataset in combined_dataset_collection_and_datasets:
+            if (dataset_collection == combined_dataset_collection_and_dataset["dataset_collection"] and
+                    dataset_id == combined_dataset_collection_and_dataset["dataset_id"] and
+                    dataset_version == str(combined_dataset_collection_and_dataset["dataset_version"]) and
+                    dataset_name == combined_dataset_collection_and_dataset["dataset_name"]):
path: dags/synapse-dataset-to-croissant.py
user: login: BWMac
body: NIT: This way might be a bit more readable
```suggestion
            if all([
                dataset_collection == combined_dataset_collection_and_dataset["dataset_collection"],
                dataset_id == combined_dataset_collection_and_dataset["dataset_id"],
                dataset_version == str(combined_dataset_collection_and_dataset["dataset_version"]),
                dataset_name == combined_dataset_collection_and_dataset["dataset_name"]
            ]):
```
created_at: 2025-04-07T18:18:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,552 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
+        elif not file_md5s.empty and int(file.id.replace("syn", "")) in file_md5s["ID"].values:
+            file_md5 = file_md5s.loc[file_md5s["ID"] == int(
+                file.id.replace("syn", "")), "CONTENTMD5"].values[0]
+        else:
+            file_md5 = "unknown_md5"
+
+        distribution_files.append(
+            {
+                "@type": "FileObject",
+                "@id": f"{file.id}.{file.version_number}",
+                "name": f"{file.name}",
+                "description": f"Data file associated with {file.name}",
+                "contentUrl": f"https://www.synapse.org/Synapse:{file.id}.{file.version_number}",
+                "encodingFormat": "application/json",
+                "md5": file_md5,
+                "sha256": "unknown",
+            }
+        )
+    return distribution_files
+
+
+def construct_record_set_section_for_files(files_attached_to_dataset: List[File]) -> Dict[str, str]:
+    """
+    Construct the record set section for the files attached to the dataset. This is used
+    to extract the keys from the annotations of the files and create a Field for each
+    key.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+
+    Returns:
+        The record set section for the files attached to the dataset.
+    """
+    unique_annotation_keys = set()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        file_annotations = file.annotations.keys()
+        for key in file_annotations:
+            unique_annotation_keys.add(key)
+
+    metadata_fields = []
+    # Sort the set of unique_annotation_keys ignoring case, but keep the case in the result
+    unique_annotation_keys = sorted(unique_annotation_keys, key=str.casefold)
+
+    for key in unique_annotation_keys:
+        metadata_fields.append(
+            {
+                "@type": "Field",
+                "@id": f"metadata/{key}",
+                "name": key,
+                "description": "",
+                "dataType": "sc:Text",
+                "source": {
+                    "fileObject": {"@id": "metadata"},
+                    "extract": {"column": key}
+                }
+            }
+        )
+
+    return {
+        "@type": "RecordSet",
+        "@id": "default",
+        "name": "default",
+        "description": "Metadata for the dataset",
+        "field": metadata_fields
+    }
+
+
+def extract_s3_objects_to_delete(bucket_objects: List[str], dataset_collections_to_consider_for_deletion: List[str], combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the S3 objects to delete from the S3 bucket. This is done by retrieving
+    all of the objects in the S3 bucket and checking if the object matches one of the
+    datasets that will be kept after the full DAG as been run. Every object in the S3
+    bucket that matches the pattern `_syn####.###_` and
+    `_datasetCollection_syn####_` will be considered for deletion. The DAG param
+    `dataset_collections_for_cleanup` is used to filter the objects that will be
+    considered for deletion as the second filter.
+
+    The last filter enforces that each object in the S3 bucket must match a dataset
+    id, dataset version, dataset collection, and dataset name in the
+    `combined_dataset_collection_and_datasets` list. If the object does not match
+    any of the objects in the list, then it will be marked for deletion.
+
+    Arguments:
+        bucket_objects: The list of objects in the S3 bucket.
+        dataset_collections_to_consider_for_deletion: The dataset collections to
+            consider for deletion. This is passed in from the
+            `dataset_collections_for_cleanup` DAG parameter.
+    """
+    objects_to_delete = []
+
+    if not bucket_objects:
+        otel_logger.info(
+            "No objects found in S3.")
+        return objects_to_delete
+
+    for bucket_object in bucket_objects:
+        dataset_id_and_version = re.search(
+            DATASET_ID_VERSION_PATTERN, bucket_object)
+        dataset_collection = re.search(
+            DATASET_COLLECTION_PATTERN, bucket_object)
+
+        if not dataset_id_and_version or not dataset_collection:
+            otel_logger.info(
+                f"Object {bucket_object} does not match the pattern. Skipping.")
+            continue
+
+        dataset_collection = dataset_collection.group(0).replace(
+            "_datasetCollection_", "").replace("_", "")
+
+        if dataset_collection not in dataset_collections_to_consider_for_deletion:
+            otel_logger.info(
+                f"Object {bucket_object} does not match a dataset collection present in `dataset_collections_for_cleanup`. Skipping.")
+            continue
+
+        dataset_id, dataset_version = dataset_id_and_version.group(0).split(
+            ".")
+        dataset_id = dataset_id.replace("_", "")
+        dataset_version = dataset_version.replace("_", "")
+        dataset_name = bucket_object.split(dataset_id_and_version.group(0))[0]
+
+        match_found_for_object = False
+
+        for combined_dataset_collection_and_dataset in combined_dataset_collection_and_datasets:
+            if (dataset_collection == combined_dataset_collection_and_dataset["dataset_collection"] and
+                    dataset_id == combined_dataset_collection_and_dataset["dataset_id"] and
+                    dataset_version == str(combined_dataset_collection_and_dataset["dataset_version"]) and
+                    dataset_name == combined_dataset_collection_and_dataset["dataset_name"]):
+                match_found_for_object = True
+                break
+        if not match_found_for_object:
+            objects_to_delete.append(bucket_object)
+
+    return objects_to_delete
+
+
+def extract_synapse_rows_to_delete(synapse_rows: DataFrame, combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the rows to delete from the Synapse table. This is done by retrieving the
+    dataset ids along with their versions from the dataset collections passed in via
+    the `dataset_collections_for_cleanup` DAG parameter. The rows to delete are
+    determined by checking if the dataset id and version are present in the
+    `combined_dataset_collection_and_datasets` list. If the dataset id and version
+    are not present in the list, then the row is marked for deletion.
+
+    Arguments:
+        synapse_rows: A Pandas DataFrame with the columns `dataset`, `dataset_version`,
+            `dataset_collection`, and `ROW_ID`.
+        combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+            dataset collections, and dataset versions to use to determine which
+            datasets to delete.
+
+    Returns:
+        The list of rows to delete from the Synapse table.
+    """
+    rows_to_delete = []
+
+    if synapse_rows.empty:
+        otel_logger.info(
+            "No rows found in Synapse.")
+        return rows_to_delete
+
+    for synapse_row in synapse_rows.itertuples():
+        dataset_collection = synapse_row.dataset_collection
+        dataset_id = synapse_row.dataset
+        dataset_version = synapse_row.dataset_version
+        row_id = synapse_row.ROW_ID
+
+        match_found_for_object = False
+
+        for combined_dataset_collection_and_dataset in combined_dataset_collection_and_datasets:
+            if (dataset_collection == combined_dataset_collection_and_dataset["dataset_collection"] and
+                    dataset_id == combined_dataset_collection_and_dataset["dataset_id"] and
+                    dataset_version == combined_dataset_collection_and_dataset["dataset_version"]):
path: dags/synapse-dataset-to-croissant.py
user: login: BWMac
body: Same as above
created_at: 2025-04-07T18:18:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,552 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
+        elif not file_md5s.empty and int(file.id.replace("syn", "")) in file_md5s["ID"].values:
+            file_md5 = file_md5s.loc[file_md5s["ID"] == int(
+                file.id.replace("syn", "")), "CONTENTMD5"].values[0]
+        else:
+            file_md5 = "unknown_md5"
+
+        distribution_files.append(
+            {
+                "@type": "FileObject",
+                "@id": f"{file.id}.{file.version_number}",
+                "name": f"{file.name}",
+                "description": f"Data file associated with {file.name}",
+                "contentUrl": f"https://www.synapse.org/Synapse:{file.id}.{file.version_number}",
+                "encodingFormat": "application/json",
+                "md5": file_md5,
+                "sha256": "unknown",
+            }
+        )
+    return distribution_files
+
+
+def construct_record_set_section_for_files(files_attached_to_dataset: List[File]) -> Dict[str, str]:
+    """
+    Construct the record set section for the files attached to the dataset. This is used
+    to extract the keys from the annotations of the files and create a Field for each
+    key.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+
+    Returns:
+        The record set section for the files attached to the dataset.
+    """
+    unique_annotation_keys = set()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        file_annotations = file.annotations.keys()
+        for key in file_annotations:
+            unique_annotation_keys.add(key)
+
+    metadata_fields = []
+    # Sort the set of unique_annotation_keys ignoring case, but keep the case in the result
+    unique_annotation_keys = sorted(unique_annotation_keys, key=str.casefold)
+
+    for key in unique_annotation_keys:
+        metadata_fields.append(
+            {
+                "@type": "Field",
+                "@id": f"metadata/{key}",
+                "name": key,
+                "description": "",
+                "dataType": "sc:Text",
+                "source": {
+                    "fileObject": {"@id": "metadata"},
+                    "extract": {"column": key}
+                }
+            }
+        )
+
+    return {
+        "@type": "RecordSet",
+        "@id": "default",
+        "name": "default",
+        "description": "Metadata for the dataset",
+        "field": metadata_fields
+    }
+
+
+def extract_s3_objects_to_delete(bucket_objects: List[str], dataset_collections_to_consider_for_deletion: List[str], combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the S3 objects to delete from the S3 bucket. This is done by retrieving
+    all of the objects in the S3 bucket and checking if the object matches one of the
+    datasets that will be kept after the full DAG as been run. Every object in the S3
+    bucket that matches the pattern `_syn####.###_` and
+    `_datasetCollection_syn####_` will be considered for deletion. The DAG param
+    `dataset_collections_for_cleanup` is used to filter the objects that will be
+    considered for deletion as the second filter.
+
+    The last filter enforces that each object in the S3 bucket must match a dataset
+    id, dataset version, dataset collection, and dataset name in the
+    `combined_dataset_collection_and_datasets` list. If the object does not match
+    any of the objects in the list, then it will be marked for deletion.
+
+    Arguments:
+        bucket_objects: The list of objects in the S3 bucket.
+        dataset_collections_to_consider_for_deletion: The dataset collections to
+            consider for deletion. This is passed in from the
+            `dataset_collections_for_cleanup` DAG parameter.
+    """
+    objects_to_delete = []
+
+    if not bucket_objects:
+        otel_logger.info(
+            "No objects found in S3.")
+        return objects_to_delete
+
+    for bucket_object in bucket_objects:
+        dataset_id_and_version = re.search(
+            DATASET_ID_VERSION_PATTERN, bucket_object)
+        dataset_collection = re.search(
+            DATASET_COLLECTION_PATTERN, bucket_object)
+
+        if not dataset_id_and_version or not dataset_collection:
+            otel_logger.info(
+                f"Object {bucket_object} does not match the pattern. Skipping.")
+            continue
+
+        dataset_collection = dataset_collection.group(0).replace(
+            "_datasetCollection_", "").replace("_", "")
+
+        if dataset_collection not in dataset_collections_to_consider_for_deletion:
+            otel_logger.info(
+                f"Object {bucket_object} does not match a dataset collection present in `dataset_collections_for_cleanup`. Skipping.")
+            continue
+
+        dataset_id, dataset_version = dataset_id_and_version.group(0).split(
+            ".")
+        dataset_id = dataset_id.replace("_", "")
+        dataset_version = dataset_version.replace("_", "")
+        dataset_name = bucket_object.split(dataset_id_and_version.group(0))[0]
+
+        match_found_for_object = False
+
+        for combined_dataset_collection_and_dataset in combined_dataset_collection_and_datasets:
+            if (dataset_collection == combined_dataset_collection_and_dataset["dataset_collection"] and
+                    dataset_id == combined_dataset_collection_and_dataset["dataset_id"] and
+                    dataset_version == str(combined_dataset_collection_and_dataset["dataset_version"]) and
+                    dataset_name == combined_dataset_collection_and_dataset["dataset_name"]):
+                match_found_for_object = True
+                break
+        if not match_found_for_object:
+            objects_to_delete.append(bucket_object)
+
+    return objects_to_delete
+
+
+def extract_synapse_rows_to_delete(synapse_rows: DataFrame, combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the rows to delete from the Synapse table. This is done by retrieving the
+    dataset ids along with their versions from the dataset collections passed in via
+    the `dataset_collections_for_cleanup` DAG parameter. The rows to delete are
+    determined by checking if the dataset id and version are present in the
+    `combined_dataset_collection_and_datasets` list. If the dataset id and version
+    are not present in the list, then the row is marked for deletion.
+
+    Arguments:
+        synapse_rows: A Pandas DataFrame with the columns `dataset`, `dataset_version`,
+            `dataset_collection`, and `ROW_ID`.
+        combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+            dataset collections, and dataset versions to use to determine which
+            datasets to delete.
+
+    Returns:
+        The list of rows to delete from the Synapse table.
+    """
+    rows_to_delete = []
+
+    if synapse_rows.empty:
+        otel_logger.info(
+            "No rows found in Synapse.")
+        return rows_to_delete
+
+    for synapse_row in synapse_rows.itertuples():
+        dataset_collection = synapse_row.dataset_collection
+        dataset_id = synapse_row.dataset
+        dataset_version = synapse_row.dataset_version
+        row_id = synapse_row.ROW_ID
+
+        match_found_for_object = False
+
+        for combined_dataset_collection_and_dataset in combined_dataset_collection_and_datasets:
+            if (dataset_collection == combined_dataset_collection_and_dataset["dataset_collection"] and
+                    dataset_id == combined_dataset_collection_and_dataset["dataset_id"] and
+                    dataset_version == combined_dataset_collection_and_dataset["dataset_version"]):
+                match_found_for_object = True
+                break
+        if not match_found_for_object:
+            rows_to_delete.append(row_id)
+
+    return rows_to_delete
+
+
+def execute_push_to_s3(dataset: Entity, dataset_id: str, dataset_version: str, s3_key: str, croissant_file: Dict[str, Any], push_to_s3: bool, **context) -> None:
+    """
+    Handle the push to S3 of the croissant file. This is done by using the S3Hook to
+    upload the file to S3. The S3 bucket is stored in the `org-sagebase-dpe-prod` AWS
+    account.
+
+    Arguments:
+        dataset: The dataset to push to S3.
+        dataset_id: The ID of the dataset.
+        dataset_version: The version of the dataset.
+        s3_key: The S3 key to use to push the file to S3.
+        croissant_file: The croissant file to push to S3.
+        push_to_s3: A boolean to indicate if the results should be pushed to S3.
+            When set to `False`, the results will be printed to the logs.
+        context: The context of the DAG run.
+
+    Returns:
+
+    """
+    try:
+        if not push_to_s3:
+            otel_logger.info(
+                f"Croissant file for [dataset: {dataset.name}, id: {dataset_id}.{dataset_version}]:\n{json.dumps(croissant_file)}")
+            return
+
+        otel_logger.info(
+            f"Uploading croissant file for [dataset: {dataset.name}, id: {dataset_id}.{dataset_version}]")
+
+        croissant_metadata_bytes = json.dumps(croissant_file).encode(
+            'utf-8')
+        metadata_file = BytesIO(croissant_metadata_bytes)
+        s3_hook = S3Hook(
+            aws_conn_id=context["params"]["aws_conn_id"], region_name=REGION_NAME, extra_args={
+                "ContentType": "application/ld+json"
+            }
+        )
+
+        otel_logger.info(
+            f"Uploading croissant file to S3: {s3_key}")
+        s3_hook.load_file_obj(file_obj=metadata_file,
+                              key=s3_key,
+                              bucket_name=BUCKET_NAME,
+                              replace=True,
+                              )
+    except Exception as ex:
+        otel_logger.exception(
+            "Failed to query snowflake and push croissant file to S3.")
+        otel_tracer.span_processor.force_flush()
+        otel_logger.handlers[0].flush()
+        raise ex
+
+
+def execute_push_to_synapse(push_to_synapse: bool, dataset: Entity, dataset_id: str, dataset_version: str, dataset_collection: str, s3_url: str, **context) -> None:
+    """
+    Handle the push to Synapse of the croissant file link. This is done by using
+    an unauthenticated Synapse client to first query the table to determine if an
+    update is needed. If the link already exists with the expected S3 URL, then
+    skip the update. If the link does not exist or the S3 URL is different, then
+    update the link with the new S3 URL using the authenticated Synapse client.
+
+    Arguments:
+        push_to_synapse: A boolean to indicate if the results should be pushed to
+            Synapse. When set to `False`, the results will be printed to the logs.
+        dataset: The dataset to push to Synapse.
+        dataset_id: The ID of the dataset.
+        dataset_version: The version of the dataset.
+        dataset_collection: The ID of the dataset collection where the dataset came from.
+        s3_url: The S3 URL to use for the value of the cell in the table.
+        syn_client: The unauthenticated Synapse client to use to query the table.
+        context: The context of the DAG run.
+
+    Returns:
+        None
+    """
+    try:
+        if not push_to_synapse:
+            otel_logger.info(
+                f"Croissant file link for [dataset: {dataset.name}, id: {dataset_id}.{dataset_version}]: {s3_url}")
+            return
+
+        otel_logger.info(
+            f"Uploading croissant file link to Synapse table {SYNAPSE_TABLE_FOR_CROISSANT_LINKS}"
+        )
+
+        # TODO: When 4.8.0 of the SYNPY client is released this may be
+        # replaced by the upsert functionality
+
+        # Warning: Using an authenticated Synapse Client during this section of code
+        syn_hook = SynapseHook(
+            context["params"]["synapse_conn_id"])
+        authenticated_syn_client: Synapse = syn_hook.client
+        authenticated_syn_client._rest_call = MethodType(
+            _rest_call_replacement, authenticated_syn_client)
+        existing_row = authenticated_syn_client.tableQuery(
+            query=f"SELECT * FROM {SYNAPSE_TABLE_FOR_CROISSANT_LINKS} WHERE dataset = '{dataset_id}' AND dataset_version = {dataset_version} AND dataset_collection = '{dataset_collection}'", resultsAs="csv")
+        existing_row_df = existing_row.asDataFrame()
+        os.remove(existing_row.filepath)
path: dags/synapse-dataset-to-croissant.py
user: login: BWMac
body: Any concern about some of these temporary files not being `remove`-d if an error happens?
created_at: 2025-04-07T18:22:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,552 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
+        elif not file_md5s.empty and int(file.id.replace("syn", "")) in file_md5s["ID"].values:
+            file_md5 = file_md5s.loc[file_md5s["ID"] == int(
+                file.id.replace("syn", "")), "CONTENTMD5"].values[0]
+        else:
+            file_md5 = "unknown_md5"
+
+        distribution_files.append(
+            {
+                "@type": "FileObject",
+                "@id": f"{file.id}.{file.version_number}",
+                "name": f"{file.name}",
+                "description": f"Data file associated with {file.name}",
+                "contentUrl": f"https://www.synapse.org/Synapse:{file.id}.{file.version_number}",
+                "encodingFormat": "application/json",
+                "md5": file_md5,
+                "sha256": "unknown",
+            }
+        )
+    return distribution_files
+
+
+def construct_record_set_section_for_files(files_attached_to_dataset: List[File]) -> Dict[str, str]:
+    """
+    Construct the record set section for the files attached to the dataset. This is used
+    to extract the keys from the annotations of the files and create a Field for each
+    key.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+
+    Returns:
+        The record set section for the files attached to the dataset.
+    """
+    unique_annotation_keys = set()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        file_annotations = file.annotations.keys()
+        for key in file_annotations:
+            unique_annotation_keys.add(key)
+
+    metadata_fields = []
+    # Sort the set of unique_annotation_keys ignoring case, but keep the case in the result
+    unique_annotation_keys = sorted(unique_annotation_keys, key=str.casefold)
+
+    for key in unique_annotation_keys:
+        metadata_fields.append(
+            {
+                "@type": "Field",
+                "@id": f"metadata/{key}",
+                "name": key,
+                "description": "",
+                "dataType": "sc:Text",
+                "source": {
+                    "fileObject": {"@id": "metadata"},
+                    "extract": {"column": key}
+                }
+            }
+        )
+
+    return {
+        "@type": "RecordSet",
+        "@id": "default",
+        "name": "default",
+        "description": "Metadata for the dataset",
+        "field": metadata_fields
+    }
+
+
+def extract_s3_objects_to_delete(bucket_objects: List[str], dataset_collections_to_consider_for_deletion: List[str], combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the S3 objects to delete from the S3 bucket. This is done by retrieving
+    all of the objects in the S3 bucket and checking if the object matches one of the
+    datasets that will be kept after the full DAG as been run. Every object in the S3
+    bucket that matches the pattern `_syn####.###_` and
+    `_datasetCollection_syn####_` will be considered for deletion. The DAG param
+    `dataset_collections_for_cleanup` is used to filter the objects that will be
+    considered for deletion as the second filter.
+
+    The last filter enforces that each object in the S3 bucket must match a dataset
+    id, dataset version, dataset collection, and dataset name in the
+    `combined_dataset_collection_and_datasets` list. If the object does not match
+    any of the objects in the list, then it will be marked for deletion.
+
+    Arguments:
+        bucket_objects: The list of objects in the S3 bucket.
+        dataset_collections_to_consider_for_deletion: The dataset collections to
+            consider for deletion. This is passed in from the
+            `dataset_collections_for_cleanup` DAG parameter.
+    """
+    objects_to_delete = []
+
+    if not bucket_objects:
+        otel_logger.info(
+            "No objects found in S3.")
+        return objects_to_delete
+
+    for bucket_object in bucket_objects:
+        dataset_id_and_version = re.search(
+            DATASET_ID_VERSION_PATTERN, bucket_object)
+        dataset_collection = re.search(
+            DATASET_COLLECTION_PATTERN, bucket_object)
+
+        if not dataset_id_and_version or not dataset_collection:
+            otel_logger.info(
+                f"Object {bucket_object} does not match the pattern. Skipping.")
+            continue
+
+        dataset_collection = dataset_collection.group(0).replace(
+            "_datasetCollection_", "").replace("_", "")
+
+        if dataset_collection not in dataset_collections_to_consider_for_deletion:
+            otel_logger.info(
+                f"Object {bucket_object} does not match a dataset collection present in `dataset_collections_for_cleanup`. Skipping.")
+            continue
+
+        dataset_id, dataset_version = dataset_id_and_version.group(0).split(
+            ".")
+        dataset_id = dataset_id.replace("_", "")
+        dataset_version = dataset_version.replace("_", "")
+        dataset_name = bucket_object.split(dataset_id_and_version.group(0))[0]
+
+        match_found_for_object = False
+
+        for combined_dataset_collection_and_dataset in combined_dataset_collection_and_datasets:
+            if (dataset_collection == combined_dataset_collection_and_dataset["dataset_collection"] and
+                    dataset_id == combined_dataset_collection_and_dataset["dataset_id"] and
+                    dataset_version == str(combined_dataset_collection_and_dataset["dataset_version"]) and
+                    dataset_name == combined_dataset_collection_and_dataset["dataset_name"]):
+                match_found_for_object = True
+                break
+        if not match_found_for_object:
+            objects_to_delete.append(bucket_object)
+
+    return objects_to_delete
+
+
+def extract_synapse_rows_to_delete(synapse_rows: DataFrame, combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the rows to delete from the Synapse table. This is done by retrieving the
+    dataset ids along with their versions from the dataset collections passed in via
+    the `dataset_collections_for_cleanup` DAG parameter. The rows to delete are
+    determined by checking if the dataset id and version are present in the
+    `combined_dataset_collection_and_datasets` list. If the dataset id and version
+    are not present in the list, then the row is marked for deletion.
+
+    Arguments:
+        synapse_rows: A Pandas DataFrame with the columns `dataset`, `dataset_version`,
+            `dataset_collection`, and `ROW_ID`.
+        combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+            dataset collections, and dataset versions to use to determine which
+            datasets to delete.
+
+    Returns:
+        The list of rows to delete from the Synapse table.
+    """
+    rows_to_delete = []
+
+    if synapse_rows.empty:
+        otel_logger.info(
+            "No rows found in Synapse.")
+        return rows_to_delete
+
+    for synapse_row in synapse_rows.itertuples():
+        dataset_collection = synapse_row.dataset_collection
+        dataset_id = synapse_row.dataset
+        dataset_version = synapse_row.dataset_version
+        row_id = synapse_row.ROW_ID
+
+        match_found_for_object = False
+
+        for combined_dataset_collection_and_dataset in combined_dataset_collection_and_datasets:
+            if (dataset_collection == combined_dataset_collection_and_dataset["dataset_collection"] and
+                    dataset_id == combined_dataset_collection_and_dataset["dataset_id"] and
+                    dataset_version == combined_dataset_collection_and_dataset["dataset_version"]):
+                match_found_for_object = True
+                break
+        if not match_found_for_object:
+            rows_to_delete.append(row_id)
+
+    return rows_to_delete
+
+
+def execute_push_to_s3(dataset: Entity, dataset_id: str, dataset_version: str, s3_key: str, croissant_file: Dict[str, Any], push_to_s3: bool, **context) -> None:
+    """
+    Handle the push to S3 of the croissant file. This is done by using the S3Hook to
+    upload the file to S3. The S3 bucket is stored in the `org-sagebase-dpe-prod` AWS
+    account.
+
+    Arguments:
+        dataset: The dataset to push to S3.
+        dataset_id: The ID of the dataset.
+        dataset_version: The version of the dataset.
+        s3_key: The S3 key to use to push the file to S3.
+        croissant_file: The croissant file to push to S3.
+        push_to_s3: A boolean to indicate if the results should be pushed to S3.
+            When set to `False`, the results will be printed to the logs.
+        context: The context of the DAG run.
+
+    Returns:
+
+    """
+    try:
+        if not push_to_s3:
+            otel_logger.info(
+                f"Croissant file for [dataset: {dataset.name}, id: {dataset_id}.{dataset_version}]:\n{json.dumps(croissant_file)}")
+            return
+
+        otel_logger.info(
+            f"Uploading croissant file for [dataset: {dataset.name}, id: {dataset_id}.{dataset_version}]")
+
+        croissant_metadata_bytes = json.dumps(croissant_file).encode(
+            'utf-8')
+        metadata_file = BytesIO(croissant_metadata_bytes)
+        s3_hook = S3Hook(
+            aws_conn_id=context["params"]["aws_conn_id"], region_name=REGION_NAME, extra_args={
+                "ContentType": "application/ld+json"
+            }
+        )
+
+        otel_logger.info(
+            f"Uploading croissant file to S3: {s3_key}")
+        s3_hook.load_file_obj(file_obj=metadata_file,
+                              key=s3_key,
+                              bucket_name=BUCKET_NAME,
+                              replace=True,
+                              )
+    except Exception as ex:
+        otel_logger.exception(
+            "Failed to query snowflake and push croissant file to S3.")
+        otel_tracer.span_processor.force_flush()
+        otel_logger.handlers[0].flush()
+        raise ex
+
+
+def execute_push_to_synapse(push_to_synapse: bool, dataset: Entity, dataset_id: str, dataset_version: str, dataset_collection: str, s3_url: str, **context) -> None:
+    """
+    Handle the push to Synapse of the croissant file link. This is done by using
+    an unauthenticated Synapse client to first query the table to determine if an
path: dags/synapse-dataset-to-croissant.py
user: login: SageGJ
body: It looks like this method is currently using an authenticated client
created_at: 2025-04-07T18:35:02Z
author_association: NONE
diff_hunk: @@ -165,17 +199,552 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
path: dags/synapse-dataset-to-croissant.py
user: login: BryanFauble
body: I couldn't find any limit on the number of placeholders that may be used. I did find limits that the query length can be up to 1 MB, which, I don't think we would hit even if we were using a dataset that had 30k entities on it (It'd be 90k placeholder strings). It would only be around 500 KB
created_at: 2025-04-07T18:36:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,552 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
path: dags/synapse-dataset-to-croissant.py
user: login: thomasyu888
body: Maybe premature simplification but would it be more simple to just pull everything from snowflake instead?

(Future ticket if yes)
created_at: 2025-04-07T18:40:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,552 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
+        elif not file_md5s.empty and int(file.id.replace("syn", "")) in file_md5s["ID"].values:
+            file_md5 = file_md5s.loc[file_md5s["ID"] == int(
+                file.id.replace("syn", "")), "CONTENTMD5"].values[0]
+        else:
+            file_md5 = "unknown_md5"
+
+        distribution_files.append(
+            {
+                "@type": "FileObject",
+                "@id": f"{file.id}.{file.version_number}",
+                "name": f"{file.name}",
+                "description": f"Data file associated with {file.name}",
+                "contentUrl": f"https://www.synapse.org/Synapse:{file.id}.{file.version_number}",
+                "encodingFormat": "application/json",
+                "md5": file_md5,
+                "sha256": "unknown",
+            }
+        )
+    return distribution_files
+
+
+def construct_record_set_section_for_files(files_attached_to_dataset: List[File]) -> Dict[str, str]:
+    """
+    Construct the record set section for the files attached to the dataset. This is used
+    to extract the keys from the annotations of the files and create a Field for each
+    key.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+
+    Returns:
+        The record set section for the files attached to the dataset.
+    """
+    unique_annotation_keys = set()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        file_annotations = file.annotations.keys()
+        for key in file_annotations:
+            unique_annotation_keys.add(key)
+
+    metadata_fields = []
+    # Sort the set of unique_annotation_keys ignoring case, but keep the case in the result
+    unique_annotation_keys = sorted(unique_annotation_keys, key=str.casefold)
+
+    for key in unique_annotation_keys:
+        metadata_fields.append(
+            {
+                "@type": "Field",
+                "@id": f"metadata/{key}",
+                "name": key,
+                "description": "",
+                "dataType": "sc:Text",
+                "source": {
+                    "fileObject": {"@id": "metadata"},
+                    "extract": {"column": key}
+                }
+            }
+        )
+
+    return {
+        "@type": "RecordSet",
+        "@id": "default",
+        "name": "default",
+        "description": "Metadata for the dataset",
+        "field": metadata_fields
+    }
+
+
+def extract_s3_objects_to_delete(bucket_objects: List[str], dataset_collections_to_consider_for_deletion: List[str], combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the S3 objects to delete from the S3 bucket. This is done by retrieving
+    all of the objects in the S3 bucket and checking if the object matches one of the
+    datasets that will be kept after the full DAG as been run. Every object in the S3
+    bucket that matches the pattern `_syn####.###_` and
+    `_datasetCollection_syn####_` will be considered for deletion. The DAG param
+    `dataset_collections_for_cleanup` is used to filter the objects that will be
+    considered for deletion as the second filter.
+
+    The last filter enforces that each object in the S3 bucket must match a dataset
+    id, dataset version, dataset collection, and dataset name in the
+    `combined_dataset_collection_and_datasets` list. If the object does not match
+    any of the objects in the list, then it will be marked for deletion.
+
+    Arguments:
+        bucket_objects: The list of objects in the S3 bucket.
+        dataset_collections_to_consider_for_deletion: The dataset collections to
+            consider for deletion. This is passed in from the
+            `dataset_collections_for_cleanup` DAG parameter.
+    """
+    objects_to_delete = []
+
+    if not bucket_objects:
+        otel_logger.info(
+            "No objects found in S3.")
+        return objects_to_delete
+
+    for bucket_object in bucket_objects:
+        dataset_id_and_version = re.search(
+            DATASET_ID_VERSION_PATTERN, bucket_object)
+        dataset_collection = re.search(
+            DATASET_COLLECTION_PATTERN, bucket_object)
+
+        if not dataset_id_and_version or not dataset_collection:
+            otel_logger.info(
+                f"Object {bucket_object} does not match the pattern. Skipping.")
+            continue
+
+        dataset_collection = dataset_collection.group(0).replace(
+            "_datasetCollection_", "").replace("_", "")
+
+        if dataset_collection not in dataset_collections_to_consider_for_deletion:
+            otel_logger.info(
+                f"Object {bucket_object} does not match a dataset collection present in `dataset_collections_for_cleanup`. Skipping.")
+            continue
+
+        dataset_id, dataset_version = dataset_id_and_version.group(0).split(
+            ".")
+        dataset_id = dataset_id.replace("_", "")
+        dataset_version = dataset_version.replace("_", "")
+        dataset_name = bucket_object.split(dataset_id_and_version.group(0))[0]
+
+        match_found_for_object = False
+
+        for combined_dataset_collection_and_dataset in combined_dataset_collection_and_datasets:
+            if (dataset_collection == combined_dataset_collection_and_dataset["dataset_collection"] and
+                    dataset_id == combined_dataset_collection_and_dataset["dataset_id"] and
+                    dataset_version == str(combined_dataset_collection_and_dataset["dataset_version"]) and
+                    dataset_name == combined_dataset_collection_and_dataset["dataset_name"]):
+                match_found_for_object = True
+                break
+        if not match_found_for_object:
+            objects_to_delete.append(bucket_object)
+
+    return objects_to_delete
+
+
+def extract_synapse_rows_to_delete(synapse_rows: DataFrame, combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the rows to delete from the Synapse table. This is done by retrieving the
+    dataset ids along with their versions from the dataset collections passed in via
+    the `dataset_collections_for_cleanup` DAG parameter. The rows to delete are
+    determined by checking if the dataset id and version are present in the
+    `combined_dataset_collection_and_datasets` list. If the dataset id and version
+    are not present in the list, then the row is marked for deletion.
+
+    Arguments:
+        synapse_rows: A Pandas DataFrame with the columns `dataset`, `dataset_version`,
+            `dataset_collection`, and `ROW_ID`.
+        combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+            dataset collections, and dataset versions to use to determine which
+            datasets to delete.
+
+    Returns:
+        The list of rows to delete from the Synapse table.
+    """
+    rows_to_delete = []
+
+    if synapse_rows.empty:
+        otel_logger.info(
+            "No rows found in Synapse.")
+        return rows_to_delete
+
+    for synapse_row in synapse_rows.itertuples():
+        dataset_collection = synapse_row.dataset_collection
+        dataset_id = synapse_row.dataset
+        dataset_version = synapse_row.dataset_version
+        row_id = synapse_row.ROW_ID
+
+        match_found_for_object = False
+
+        for combined_dataset_collection_and_dataset in combined_dataset_collection_and_datasets:
+            if (dataset_collection == combined_dataset_collection_and_dataset["dataset_collection"] and
+                    dataset_id == combined_dataset_collection_and_dataset["dataset_id"] and
+                    dataset_version == combined_dataset_collection_and_dataset["dataset_version"]):
+                match_found_for_object = True
+                break
+        if not match_found_for_object:
+            rows_to_delete.append(row_id)
+
+    return rows_to_delete
+
+
+def execute_push_to_s3(dataset: Entity, dataset_id: str, dataset_version: str, s3_key: str, croissant_file: Dict[str, Any], push_to_s3: bool, **context) -> None:
+    """
+    Handle the push to S3 of the croissant file. This is done by using the S3Hook to
+    upload the file to S3. The S3 bucket is stored in the `org-sagebase-dpe-prod` AWS
+    account.
+
+    Arguments:
+        dataset: The dataset to push to S3.
+        dataset_id: The ID of the dataset.
+        dataset_version: The version of the dataset.
+        s3_key: The S3 key to use to push the file to S3.
+        croissant_file: The croissant file to push to S3.
+        push_to_s3: A boolean to indicate if the results should be pushed to S3.
+            When set to `False`, the results will be printed to the logs.
+        context: The context of the DAG run.
+
+    Returns:
+
+    """
+    try:
+        if not push_to_s3:
+            otel_logger.info(
+                f"Croissant file for [dataset: {dataset.name}, id: {dataset_id}.{dataset_version}]:\n{json.dumps(croissant_file)}")
+            return
+
+        otel_logger.info(
+            f"Uploading croissant file for [dataset: {dataset.name}, id: {dataset_id}.{dataset_version}]")
+
+        croissant_metadata_bytes = json.dumps(croissant_file).encode(
+            'utf-8')
+        metadata_file = BytesIO(croissant_metadata_bytes)
+        s3_hook = S3Hook(
+            aws_conn_id=context["params"]["aws_conn_id"], region_name=REGION_NAME, extra_args={
+                "ContentType": "application/ld+json"
+            }
+        )
+
+        otel_logger.info(
+            f"Uploading croissant file to S3: {s3_key}")
+        s3_hook.load_file_obj(file_obj=metadata_file,
+                              key=s3_key,
+                              bucket_name=BUCKET_NAME,
+                              replace=True,
+                              )
+    except Exception as ex:
+        otel_logger.exception(
+            "Failed to query snowflake and push croissant file to S3.")
+        otel_tracer.span_processor.force_flush()
+        otel_logger.handlers[0].flush()
+        raise ex
+
+
+def execute_push_to_synapse(push_to_synapse: bool, dataset: Entity, dataset_id: str, dataset_version: str, dataset_collection: str, s3_url: str, **context) -> None:
+    """
+    Handle the push to Synapse of the croissant file link. This is done by using
+    an unauthenticated Synapse client to first query the table to determine if an
+    update is needed. If the link already exists with the expected S3 URL, then
+    skip the update. If the link does not exist or the S3 URL is different, then
+    update the link with the new S3 URL using the authenticated Synapse client.
+
+    Arguments:
+        push_to_synapse: A boolean to indicate if the results should be pushed to
+            Synapse. When set to `False`, the results will be printed to the logs.
+        dataset: The dataset to push to Synapse.
+        dataset_id: The ID of the dataset.
+        dataset_version: The version of the dataset.
+        dataset_collection: The ID of the dataset collection where the dataset came from.
+        s3_url: The S3 URL to use for the value of the cell in the table.
+        syn_client: The unauthenticated Synapse client to use to query the table.
+        context: The context of the DAG run.
+
+    Returns:
+        None
+    """
+    try:
+        if not push_to_synapse:
+            otel_logger.info(
+                f"Croissant file link for [dataset: {dataset.name}, id: {dataset_id}.{dataset_version}]: {s3_url}")
+            return
+
+        otel_logger.info(
+            f"Uploading croissant file link to Synapse table {SYNAPSE_TABLE_FOR_CROISSANT_LINKS}"
+        )
+
+        # TODO: When 4.8.0 of the SYNPY client is released this may be
+        # replaced by the upsert functionality
+
+        # Warning: Using an authenticated Synapse Client during this section of code
+        syn_hook = SynapseHook(
+            context["params"]["synapse_conn_id"])
+        authenticated_syn_client: Synapse = syn_hook.client
+        authenticated_syn_client._rest_call = MethodType(
+            _rest_call_replacement, authenticated_syn_client)
+        existing_row = authenticated_syn_client.tableQuery(
+            query=f"SELECT * FROM {SYNAPSE_TABLE_FOR_CROISSANT_LINKS} WHERE dataset = '{dataset_id}' AND dataset_version = {dataset_version} AND dataset_collection = '{dataset_collection}'", resultsAs="csv")
+        existing_row_df = existing_row.asDataFrame()
+        os.remove(existing_row.filepath)
+        if not existing_row_df.empty and existing_row_df["croissant_file_s3_object"].values[0] == s3_url:
+            otel_logger.info(
+                f"Croissant file link already exists in Synapse table {SYNAPSE_TABLE_FOR_CROISSANT_LINKS}. Skipping.")
+            return
+
+        df = DataFrame(
+            data={
+                "dataset": [dataset_id],
+                "dataset_version": [dataset_version],
+                "dataset_collection": [dataset_collection],
+                "croissant_file_s3_object": [s3_url]
+            }
+        )
+        temp_dir = tempfile.mkdtemp()
+        filepath = os.path.join(temp_dir, "table.csv")
+
+        schema = authenticated_syn_client.get(
+            SYNAPSE_TABLE_FOR_CROISSANT_LINKS)
+        if existing_row_df.empty:
+            # If the row does not exist, create a new row
+            authenticated_syn_client.store(
+                Table(schema=schema, values=df, filepath=filepath))
+        else:
+            # Update the existing row with the new value
+            existing_row_df["croissant_file_s3_object"] = [s3_url]
+            authenticated_syn_client.store(
+                Table(schema=schema, values=existing_row_df, filepath=filepath))
+
+        os.remove(filepath)
+    except Exception as ex:
+        otel_logger.exception(
+            "Failed to push croissant file link to Synapse.")
+        otel_tracer.span_processor.force_flush()
+        otel_logger.handlers[0].flush()
+        raise ex
+
+
 @dag(**dag_config)
 def dataset_to_croissant() -> None:
-    """Execute a query on Snowflake and report the results to a Synapse table."""
+    """Execute a query on Snowflake and report the results to a Synapse table.
+
+
+    DAG Parameters:
path: dags/synapse-dataset-to-croissant.py
user: login: thomasyu888
body: This, imo, is a good example of PR review comment that doesn't have to be in the code comment itself 
created_at: 2025-04-07T18:41:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -292,272 +867,277 @@ def combine_dataset_lists(root_carrier_context: Dict, dataset_ids: List[List[Dic
         Returns:
             The combined list of dataset IDs.
         """
-        with otel_tracer.start_as_current_span("combine_dataset_lists", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+        with otel_tracer.start_as_current_span("combine_dataset_lists", context=TraceContextTextMapPropagator().extract(root_carrier_context)):
             return_value = [
                 dataset for datasets in dataset_ids for dataset in datasets]
         otel_tracer.span_processor.force_flush()
         return return_value
 
     @task
-    def query_snowflake_and_push_croissant_file(root_carrier_context: Dict, dataset_id: str, **context) -> None:
+    def delete_non_current_croissant_file_in_s3(
+        root_carrier_context: Dict, combined_dataset_collection_and_datasets: List[Dict[str, str]], **context
+    ) -> None:
+        """
+        Delete the non-current croissant files from S3. 
+        This is used to remove the old files from S3 that are no longer needed. 
+        A "non-current" file is defined as a croissant JSON LD file which is no longer
+        present in any Dataset collection.
+
+        This can occur if the dataset has been removed from the dataset collection, or
+        if there is a new version of the dataset that has been added to the dataset
+        collection.
+
+        Arguments:
+            root_carrier_context: The root carrier context to use for the trace context.
+            combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+                dataset collections, and dataset versions to use to determine which
+                datasets to delete.
+
+        Returns:
+            None
+        """
+        with otel_tracer.start_as_current_span("delete_non_current_files_from_s3", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            s3_hook = S3Hook(
+                aws_conn_id=context["params"]["aws_conn_id"], region_name=REGION_NAME)
+            bucket_objects = s3_hook.list_keys(bucket_name=BUCKET_NAME)
+
+            objects_to_delete = extract_s3_objects_to_delete(
+                bucket_objects=bucket_objects,
+                dataset_collections_to_consider_for_deletion=context[
+                    "params"]["dataset_collections_for_cleanup"],
+                combined_dataset_collection_and_datasets=combined_dataset_collection_and_datasets,
+            )
+
+            if objects_to_delete:
+                delete_out_of_date_from_s3 = context["params"]["delete_out_of_date_from_s3"]
+                if delete_out_of_date_from_s3:
+                    otel_logger.info(
+                        f"Deleting the following objects from S3: {objects_to_delete}")
+                    s3_hook.delete_objects(
+                        bucket=BUCKET_NAME, keys=objects_to_delete)
+                else:
+                    otel_logger.info(
path: dags/synapse-dataset-to-croissant.py
user: login: SageGJ
body: What would cause the objects to not be deleted here?
created_at: 2025-04-07T18:46:45Z
author_association: NONE
diff_hunk: @@ -292,272 +867,277 @@ def combine_dataset_lists(root_carrier_context: Dict, dataset_ids: List[List[Dic
         Returns:
             The combined list of dataset IDs.
         """
-        with otel_tracer.start_as_current_span("combine_dataset_lists", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+        with otel_tracer.start_as_current_span("combine_dataset_lists", context=TraceContextTextMapPropagator().extract(root_carrier_context)):
             return_value = [
                 dataset for datasets in dataset_ids for dataset in datasets]
         otel_tracer.span_processor.force_flush()
         return return_value
 
     @task
-    def query_snowflake_and_push_croissant_file(root_carrier_context: Dict, dataset_id: str, **context) -> None:
+    def delete_non_current_croissant_file_in_s3(
+        root_carrier_context: Dict, combined_dataset_collection_and_datasets: List[Dict[str, str]], **context
+    ) -> None:
+        """
+        Delete the non-current croissant files from S3. 
+        This is used to remove the old files from S3 that are no longer needed. 
+        A "non-current" file is defined as a croissant JSON LD file which is no longer
+        present in any Dataset collection.
+
+        This can occur if the dataset has been removed from the dataset collection, or
+        if there is a new version of the dataset that has been added to the dataset
+        collection.
+
+        Arguments:
+            root_carrier_context: The root carrier context to use for the trace context.
+            combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+                dataset collections, and dataset versions to use to determine which
+                datasets to delete.
+
+        Returns:
+            None
+        """
+        with otel_tracer.start_as_current_span("delete_non_current_files_from_s3", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            s3_hook = S3Hook(
+                aws_conn_id=context["params"]["aws_conn_id"], region_name=REGION_NAME)
+            bucket_objects = s3_hook.list_keys(bucket_name=BUCKET_NAME)
+
+            objects_to_delete = extract_s3_objects_to_delete(
+                bucket_objects=bucket_objects,
+                dataset_collections_to_consider_for_deletion=context[
+                    "params"]["dataset_collections_for_cleanup"],
+                combined_dataset_collection_and_datasets=combined_dataset_collection_and_datasets,
+            )
+
+            if objects_to_delete:
+                delete_out_of_date_from_s3 = context["params"]["delete_out_of_date_from_s3"]
+                if delete_out_of_date_from_s3:
+                    otel_logger.info(
+                        f"Deleting the following objects from S3: {objects_to_delete}")
+                    s3_hook.delete_objects(
+                        bucket=BUCKET_NAME, keys=objects_to_delete)
+                else:
+                    otel_logger.info(
+                        f"Found objects to delete from S3, but not deleting due to `delete_out_of_date_from_s3` param: {objects_to_delete}")
+            else:
+                otel_logger.info(
+                    "No objects to delete from S3. All objects are current.")
+            otel_tracer.span_processor.force_flush()
+            otel_logger.handlers[0].flush()
+            return None
+
+    @task
+    def delete_non_current_croissant_file_in_synapse(
+        root_carrier_context: Dict, combined_dataset_collection_and_datasets: List[Dict[str, str]], **context
+    ) -> None:
+        """
+        Delete the non-current croissant files from the Synapse table. 
+        This is used to remove the old files from Synapse that are no longer needed. 
+        A "non-current" file is defined as a croissant JSON LD file which is no longer
+        present in any Dataset collection.
+
+        This can occur if the dataset has been removed from the dataset collection, or
+        if there is a new version of the dataset that has been added to the dataset
+        collection.
+
+        Arguments:
+            root_carrier_context: The root carrier context to use for the trace context.
+            combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+                dataset collections, and dataset versions to use to determine which
+                datasets to delete.
+
+        Returns:
+            None
+        """
+        # Warning: Using an authenticated Synapse Client during this section of code
+        with otel_tracer.start_as_current_span("delete_non_current_files_from_s3", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            syn_hook = SynapseHook(context["params"]["synapse_conn_id"])
+            syn_client: Synapse = syn_hook.client
+            syn_client._rest_call = MethodType(
+                _rest_call_replacement, syn_client)
path: dags/synapse-dataset-to-croissant.py
user: login: SageGJ
body: Since these lines are used in multiple places would it be useful to create a function like `create_syn_client` to create a client from a `SynaspseHook`, replace the appropriate method, and then return the new client?
created_at: 2025-04-07T18:51:47Z
author_association: NONE
diff_hunk: @@ -165,17 +199,552 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
path: dags/synapse-dataset-to-croissant.py
user: login: BryanFauble
body: Completed in https://github.com/Sage-Bionetworks-Workflows/orca-recipes/pull/84/files#r2031909088
created_at: 2025-04-07T19:46:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,552 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
+        elif not file_md5s.empty and int(file.id.replace("syn", "")) in file_md5s["ID"].values:
+            file_md5 = file_md5s.loc[file_md5s["ID"] == int(
+                file.id.replace("syn", "")), "CONTENTMD5"].values[0]
+        else:
+            file_md5 = "unknown_md5"
+
+        distribution_files.append(
+            {
+                "@type": "FileObject",
+                "@id": f"{file.id}.{file.version_number}",
+                "name": f"{file.name}",
+                "description": f"Data file associated with {file.name}",
+                "contentUrl": f"https://www.synapse.org/Synapse:{file.id}.{file.version_number}",
+                "encodingFormat": "application/json",
path: dags/synapse-dataset-to-croissant.py
user: login: BryanFauble
body: Done in https://github.com/Sage-Bionetworks-Workflows/orca-recipes/pull/84
created_at: 2025-04-07T19:46:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,552 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
+        elif not file_md5s.empty and int(file.id.replace("syn", "")) in file_md5s["ID"].values:
+            file_md5 = file_md5s.loc[file_md5s["ID"] == int(
+                file.id.replace("syn", "")), "CONTENTMD5"].values[0]
+        else:
+            file_md5 = "unknown_md5"
+
+        distribution_files.append(
+            {
+                "@type": "FileObject",
+                "@id": f"{file.id}.{file.version_number}",
+                "name": f"{file.name}",
+                "description": f"Data file associated with {file.name}",
+                "contentUrl": f"https://www.synapse.org/Synapse:{file.id}.{file.version_number}",
+                "encodingFormat": "application/json",
+                "md5": file_md5,
+                "sha256": "unknown",
+            }
+        )
+    return distribution_files
+
+
+def construct_record_set_section_for_files(files_attached_to_dataset: List[File]) -> Dict[str, str]:
+    """
+    Construct the record set section for the files attached to the dataset. This is used
+    to extract the keys from the annotations of the files and create a Field for each
+    key.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+
+    Returns:
+        The record set section for the files attached to the dataset.
+    """
+    unique_annotation_keys = set()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        file_annotations = file.annotations.keys()
+        for key in file_annotations:
+            unique_annotation_keys.add(key)
+
+    metadata_fields = []
+    # Sort the set of unique_annotation_keys ignoring case, but keep the case in the result
+    unique_annotation_keys = sorted(unique_annotation_keys, key=str.casefold)
+
+    for key in unique_annotation_keys:
+        metadata_fields.append(
+            {
+                "@type": "Field",
+                "@id": f"metadata/{key}",
+                "name": key,
+                "description": "",
+                "dataType": "sc:Text",
+                "source": {
+                    "fileObject": {"@id": "metadata"},
+                    "extract": {"column": key}
+                }
+            }
+        )
+
+    return {
+        "@type": "RecordSet",
+        "@id": "default",
+        "name": "default",
+        "description": "Metadata for the dataset",
+        "field": metadata_fields
+    }
+
+
+def extract_s3_objects_to_delete(bucket_objects: List[str], dataset_collections_to_consider_for_deletion: List[str], combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the S3 objects to delete from the S3 bucket. This is done by retrieving
+    all of the objects in the S3 bucket and checking if the object matches one of the
+    datasets that will be kept after the full DAG as been run. Every object in the S3
+    bucket that matches the pattern `_syn####.###_` and
+    `_datasetCollection_syn####_` will be considered for deletion. The DAG param
+    `dataset_collections_for_cleanup` is used to filter the objects that will be
+    considered for deletion as the second filter.
+
+    The last filter enforces that each object in the S3 bucket must match a dataset
+    id, dataset version, dataset collection, and dataset name in the
+    `combined_dataset_collection_and_datasets` list. If the object does not match
+    any of the objects in the list, then it will be marked for deletion.
+
+    Arguments:
+        bucket_objects: The list of objects in the S3 bucket.
+        dataset_collections_to_consider_for_deletion: The dataset collections to
+            consider for deletion. This is passed in from the
+            `dataset_collections_for_cleanup` DAG parameter.
+    """
+    objects_to_delete = []
+
+    if not bucket_objects:
+        otel_logger.info(
+            "No objects found in S3.")
+        return objects_to_delete
+
+    for bucket_object in bucket_objects:
+        dataset_id_and_version = re.search(
+            DATASET_ID_VERSION_PATTERN, bucket_object)
+        dataset_collection = re.search(
+            DATASET_COLLECTION_PATTERN, bucket_object)
+
+        if not dataset_id_and_version or not dataset_collection:
+            otel_logger.info(
+                f"Object {bucket_object} does not match the pattern. Skipping.")
+            continue
+
+        dataset_collection = dataset_collection.group(0).replace(
+            "_datasetCollection_", "").replace("_", "")
+
+        if dataset_collection not in dataset_collections_to_consider_for_deletion:
+            otel_logger.info(
+                f"Object {bucket_object} does not match a dataset collection present in `dataset_collections_for_cleanup`. Skipping.")
+            continue
+
+        dataset_id, dataset_version = dataset_id_and_version.group(0).split(
+            ".")
+        dataset_id = dataset_id.replace("_", "")
+        dataset_version = dataset_version.replace("_", "")
+        dataset_name = bucket_object.split(dataset_id_and_version.group(0))[0]
+
+        match_found_for_object = False
+
+        for combined_dataset_collection_and_dataset in combined_dataset_collection_and_datasets:
+            if (dataset_collection == combined_dataset_collection_and_dataset["dataset_collection"] and
+                    dataset_id == combined_dataset_collection_and_dataset["dataset_id"] and
+                    dataset_version == str(combined_dataset_collection_and_dataset["dataset_version"]) and
+                    dataset_name == combined_dataset_collection_and_dataset["dataset_name"]):
+                match_found_for_object = True
+                break
+        if not match_found_for_object:
+            objects_to_delete.append(bucket_object)
+
+    return objects_to_delete
+
+
+def extract_synapse_rows_to_delete(synapse_rows: DataFrame, combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the rows to delete from the Synapse table. This is done by retrieving the
+    dataset ids along with their versions from the dataset collections passed in via
+    the `dataset_collections_for_cleanup` DAG parameter. The rows to delete are
+    determined by checking if the dataset id and version are present in the
+    `combined_dataset_collection_and_datasets` list. If the dataset id and version
+    are not present in the list, then the row is marked for deletion.
+
+    Arguments:
+        synapse_rows: A Pandas DataFrame with the columns `dataset`, `dataset_version`,
+            `dataset_collection`, and `ROW_ID`.
+        combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+            dataset collections, and dataset versions to use to determine which
+            datasets to delete.
+
+    Returns:
+        The list of rows to delete from the Synapse table.
+    """
+    rows_to_delete = []
+
+    if synapse_rows.empty:
+        otel_logger.info(
+            "No rows found in Synapse.")
+        return rows_to_delete
+
+    for synapse_row in synapse_rows.itertuples():
+        dataset_collection = synapse_row.dataset_collection
+        dataset_id = synapse_row.dataset
+        dataset_version = synapse_row.dataset_version
+        row_id = synapse_row.ROW_ID
+
+        match_found_for_object = False
+
+        for combined_dataset_collection_and_dataset in combined_dataset_collection_and_datasets:
+            if (dataset_collection == combined_dataset_collection_and_dataset["dataset_collection"] and
+                    dataset_id == combined_dataset_collection_and_dataset["dataset_id"] and
+                    dataset_version == combined_dataset_collection_and_dataset["dataset_version"]):
+                match_found_for_object = True
+                break
+        if not match_found_for_object:
+            rows_to_delete.append(row_id)
+
+    return rows_to_delete
+
+
+def execute_push_to_s3(dataset: Entity, dataset_id: str, dataset_version: str, s3_key: str, croissant_file: Dict[str, Any], push_to_s3: bool, **context) -> None:
+    """
+    Handle the push to S3 of the croissant file. This is done by using the S3Hook to
+    upload the file to S3. The S3 bucket is stored in the `org-sagebase-dpe-prod` AWS
+    account.
+
+    Arguments:
+        dataset: The dataset to push to S3.
+        dataset_id: The ID of the dataset.
+        dataset_version: The version of the dataset.
+        s3_key: The S3 key to use to push the file to S3.
+        croissant_file: The croissant file to push to S3.
+        push_to_s3: A boolean to indicate if the results should be pushed to S3.
+            When set to `False`, the results will be printed to the logs.
+        context: The context of the DAG run.
+
+    Returns:
+
+    """
+    try:
+        if not push_to_s3:
+            otel_logger.info(
+                f"Croissant file for [dataset: {dataset.name}, id: {dataset_id}.{dataset_version}]:\n{json.dumps(croissant_file)}")
+            return
+
+        otel_logger.info(
+            f"Uploading croissant file for [dataset: {dataset.name}, id: {dataset_id}.{dataset_version}]")
+
+        croissant_metadata_bytes = json.dumps(croissant_file).encode(
+            'utf-8')
+        metadata_file = BytesIO(croissant_metadata_bytes)
+        s3_hook = S3Hook(
+            aws_conn_id=context["params"]["aws_conn_id"], region_name=REGION_NAME, extra_args={
+                "ContentType": "application/ld+json"
+            }
+        )
+
+        otel_logger.info(
+            f"Uploading croissant file to S3: {s3_key}")
+        s3_hook.load_file_obj(file_obj=metadata_file,
+                              key=s3_key,
+                              bucket_name=BUCKET_NAME,
+                              replace=True,
+                              )
+    except Exception as ex:
+        otel_logger.exception(
+            "Failed to query snowflake and push croissant file to S3.")
+        otel_tracer.span_processor.force_flush()
+        otel_logger.handlers[0].flush()
+        raise ex
+
+
+def execute_push_to_synapse(push_to_synapse: bool, dataset: Entity, dataset_id: str, dataset_version: str, dataset_collection: str, s3_url: str, **context) -> None:
+    """
+    Handle the push to Synapse of the croissant file link. This is done by using
+    an unauthenticated Synapse client to first query the table to determine if an
+    update is needed. If the link already exists with the expected S3 URL, then
+    skip the update. If the link does not exist or the S3 URL is different, then
+    update the link with the new S3 URL using the authenticated Synapse client.
+
+    Arguments:
+        push_to_synapse: A boolean to indicate if the results should be pushed to
+            Synapse. When set to `False`, the results will be printed to the logs.
+        dataset: The dataset to push to Synapse.
+        dataset_id: The ID of the dataset.
+        dataset_version: The version of the dataset.
+        dataset_collection: The ID of the dataset collection where the dataset came from.
+        s3_url: The S3 URL to use for the value of the cell in the table.
+        syn_client: The unauthenticated Synapse client to use to query the table.
+        context: The context of the DAG run.
+
+    Returns:
+        None
+    """
+    try:
+        if not push_to_synapse:
+            otel_logger.info(
+                f"Croissant file link for [dataset: {dataset.name}, id: {dataset_id}.{dataset_version}]: {s3_url}")
+            return
+
+        otel_logger.info(
+            f"Uploading croissant file link to Synapse table {SYNAPSE_TABLE_FOR_CROISSANT_LINKS}"
+        )
+
+        # TODO: When 4.8.0 of the SYNPY client is released this may be
+        # replaced by the upsert functionality
+
+        # Warning: Using an authenticated Synapse Client during this section of code
+        syn_hook = SynapseHook(
+            context["params"]["synapse_conn_id"])
+        authenticated_syn_client: Synapse = syn_hook.client
+        authenticated_syn_client._rest_call = MethodType(
+            _rest_call_replacement, authenticated_syn_client)
+        existing_row = authenticated_syn_client.tableQuery(
+            query=f"SELECT * FROM {SYNAPSE_TABLE_FOR_CROISSANT_LINKS} WHERE dataset = '{dataset_id}' AND dataset_version = {dataset_version} AND dataset_collection = '{dataset_collection}'", resultsAs="csv")
+        existing_row_df = existing_row.asDataFrame()
+        os.remove(existing_row.filepath)
path: dags/synapse-dataset-to-croissant.py
user: login: BryanFauble
body: I added a small try/finally in https://github.com/Sage-Bionetworks-Workflows/orca-recipes/pull/84

 I don't know what would really occur in these cases if temporary storage fills up. It might be a bridge we'll need to cross later, but I wasn't going to worry about it beyond what I had put in place so far.
created_at: 2025-04-07T19:48:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,552 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
+        elif not file_md5s.empty and int(file.id.replace("syn", "")) in file_md5s["ID"].values:
+            file_md5 = file_md5s.loc[file_md5s["ID"] == int(
+                file.id.replace("syn", "")), "CONTENTMD5"].values[0]
+        else:
+            file_md5 = "unknown_md5"
+
+        distribution_files.append(
+            {
+                "@type": "FileObject",
+                "@id": f"{file.id}.{file.version_number}",
+                "name": f"{file.name}",
+                "description": f"Data file associated with {file.name}",
+                "contentUrl": f"https://www.synapse.org/Synapse:{file.id}.{file.version_number}",
+                "encodingFormat": "application/json",
+                "md5": file_md5,
+                "sha256": "unknown",
+            }
+        )
+    return distribution_files
+
+
+def construct_record_set_section_for_files(files_attached_to_dataset: List[File]) -> Dict[str, str]:
+    """
+    Construct the record set section for the files attached to the dataset. This is used
+    to extract the keys from the annotations of the files and create a Field for each
+    key.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+
+    Returns:
+        The record set section for the files attached to the dataset.
+    """
+    unique_annotation_keys = set()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        file_annotations = file.annotations.keys()
+        for key in file_annotations:
+            unique_annotation_keys.add(key)
+
+    metadata_fields = []
+    # Sort the set of unique_annotation_keys ignoring case, but keep the case in the result
+    unique_annotation_keys = sorted(unique_annotation_keys, key=str.casefold)
+
+    for key in unique_annotation_keys:
+        metadata_fields.append(
+            {
+                "@type": "Field",
+                "@id": f"metadata/{key}",
+                "name": key,
+                "description": "",
+                "dataType": "sc:Text",
+                "source": {
+                    "fileObject": {"@id": "metadata"},
+                    "extract": {"column": key}
+                }
+            }
+        )
+
+    return {
+        "@type": "RecordSet",
+        "@id": "default",
+        "name": "default",
+        "description": "Metadata for the dataset",
+        "field": metadata_fields
+    }
+
+
+def extract_s3_objects_to_delete(bucket_objects: List[str], dataset_collections_to_consider_for_deletion: List[str], combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the S3 objects to delete from the S3 bucket. This is done by retrieving
+    all of the objects in the S3 bucket and checking if the object matches one of the
+    datasets that will be kept after the full DAG as been run. Every object in the S3
+    bucket that matches the pattern `_syn####.###_` and
+    `_datasetCollection_syn####_` will be considered for deletion. The DAG param
+    `dataset_collections_for_cleanup` is used to filter the objects that will be
+    considered for deletion as the second filter.
+
+    The last filter enforces that each object in the S3 bucket must match a dataset
+    id, dataset version, dataset collection, and dataset name in the
+    `combined_dataset_collection_and_datasets` list. If the object does not match
+    any of the objects in the list, then it will be marked for deletion.
+
+    Arguments:
+        bucket_objects: The list of objects in the S3 bucket.
+        dataset_collections_to_consider_for_deletion: The dataset collections to
+            consider for deletion. This is passed in from the
+            `dataset_collections_for_cleanup` DAG parameter.
+    """
+    objects_to_delete = []
+
+    if not bucket_objects:
+        otel_logger.info(
+            "No objects found in S3.")
+        return objects_to_delete
+
+    for bucket_object in bucket_objects:
+        dataset_id_and_version = re.search(
+            DATASET_ID_VERSION_PATTERN, bucket_object)
+        dataset_collection = re.search(
+            DATASET_COLLECTION_PATTERN, bucket_object)
+
+        if not dataset_id_and_version or not dataset_collection:
+            otel_logger.info(
+                f"Object {bucket_object} does not match the pattern. Skipping.")
+            continue
+
+        dataset_collection = dataset_collection.group(0).replace(
+            "_datasetCollection_", "").replace("_", "")
+
+        if dataset_collection not in dataset_collections_to_consider_for_deletion:
+            otel_logger.info(
+                f"Object {bucket_object} does not match a dataset collection present in `dataset_collections_for_cleanup`. Skipping.")
+            continue
+
+        dataset_id, dataset_version = dataset_id_and_version.group(0).split(
+            ".")
+        dataset_id = dataset_id.replace("_", "")
+        dataset_version = dataset_version.replace("_", "")
+        dataset_name = bucket_object.split(dataset_id_and_version.group(0))[0]
+
+        match_found_for_object = False
+
+        for combined_dataset_collection_and_dataset in combined_dataset_collection_and_datasets:
+            if (dataset_collection == combined_dataset_collection_and_dataset["dataset_collection"] and
+                    dataset_id == combined_dataset_collection_and_dataset["dataset_id"] and
+                    dataset_version == str(combined_dataset_collection_and_dataset["dataset_version"]) and
+                    dataset_name == combined_dataset_collection_and_dataset["dataset_name"]):
+                match_found_for_object = True
path: dags/synapse-dataset-to-croissant.py
user: login: SageGJ
body: The number of checks and what they all mean is difficult to intuit.
Something like
```suggestion
            dataset_collection_match = (dataset_collection == combined_dataset_collection_and_dataset["dataset_collection"])
            dataset_id_match = (dataset_id == combined_dataset_collection_and_dataset["dataset_id"])
            dataset_version_match = (dataset_version == str(combined_dataset_collection_and_dataset["dataset_version"]))                
            dataset_name_match = (dataset_name == combined_dataset_collection_and_dataset["dataset_name"])
            if all([
            dataset_collection_match ,
            dataset_id_match ,
            dataset_version_match,
            dataset_name_match ,
            ]):
                   match_found_for_object = True
```
might be easier for others to understand
created_at: 2025-04-07T20:21:27Z
author_association: NONE
diff_hunk: @@ -292,272 +867,277 @@ def combine_dataset_lists(root_carrier_context: Dict, dataset_ids: List[List[Dic
         Returns:
             The combined list of dataset IDs.
         """
-        with otel_tracer.start_as_current_span("combine_dataset_lists", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+        with otel_tracer.start_as_current_span("combine_dataset_lists", context=TraceContextTextMapPropagator().extract(root_carrier_context)):
             return_value = [
                 dataset for datasets in dataset_ids for dataset in datasets]
         otel_tracer.span_processor.force_flush()
         return return_value
 
     @task
-    def query_snowflake_and_push_croissant_file(root_carrier_context: Dict, dataset_id: str, **context) -> None:
+    def delete_non_current_croissant_file_in_s3(
+        root_carrier_context: Dict, combined_dataset_collection_and_datasets: List[Dict[str, str]], **context
+    ) -> None:
+        """
+        Delete the non-current croissant files from S3. 
+        This is used to remove the old files from S3 that are no longer needed. 
+        A "non-current" file is defined as a croissant JSON LD file which is no longer
+        present in any Dataset collection.
+
+        This can occur if the dataset has been removed from the dataset collection, or
+        if there is a new version of the dataset that has been added to the dataset
+        collection.
+
+        Arguments:
+            root_carrier_context: The root carrier context to use for the trace context.
+            combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+                dataset collections, and dataset versions to use to determine which
+                datasets to delete.
+
+        Returns:
+            None
+        """
+        with otel_tracer.start_as_current_span("delete_non_current_files_from_s3", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            s3_hook = S3Hook(
+                aws_conn_id=context["params"]["aws_conn_id"], region_name=REGION_NAME)
+            bucket_objects = s3_hook.list_keys(bucket_name=BUCKET_NAME)
+
+            objects_to_delete = extract_s3_objects_to_delete(
+                bucket_objects=bucket_objects,
+                dataset_collections_to_consider_for_deletion=context[
+                    "params"]["dataset_collections_for_cleanup"],
+                combined_dataset_collection_and_datasets=combined_dataset_collection_and_datasets,
+            )
+
+            if objects_to_delete:
+                delete_out_of_date_from_s3 = context["params"]["delete_out_of_date_from_s3"]
+                if delete_out_of_date_from_s3:
+                    otel_logger.info(
+                        f"Deleting the following objects from S3: {objects_to_delete}")
+                    s3_hook.delete_objects(
+                        bucket=BUCKET_NAME, keys=objects_to_delete)
+                else:
+                    otel_logger.info(
+                        f"Found objects to delete from S3, but not deleting due to `delete_out_of_date_from_s3` param: {objects_to_delete}")
+            else:
+                otel_logger.info(
+                    "No objects to delete from S3. All objects are current.")
+            otel_tracer.span_processor.force_flush()
+            otel_logger.handlers[0].flush()
+            return None
+
+    @task
+    def delete_non_current_croissant_file_in_synapse(
+        root_carrier_context: Dict, combined_dataset_collection_and_datasets: List[Dict[str, str]], **context
+    ) -> None:
+        """
+        Delete the non-current croissant files from the Synapse table. 
+        This is used to remove the old files from Synapse that are no longer needed. 
+        A "non-current" file is defined as a croissant JSON LD file which is no longer
+        present in any Dataset collection.
+
+        This can occur if the dataset has been removed from the dataset collection, or
+        if there is a new version of the dataset that has been added to the dataset
+        collection.
+
+        Arguments:
+            root_carrier_context: The root carrier context to use for the trace context.
+            combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+                dataset collections, and dataset versions to use to determine which
+                datasets to delete.
+
+        Returns:
+            None
+        """
+        # Warning: Using an authenticated Synapse Client during this section of code
+        with otel_tracer.start_as_current_span("delete_non_current_files_from_s3", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            syn_hook = SynapseHook(context["params"]["synapse_conn_id"])
+            syn_client: Synapse = syn_hook.client
+            syn_client._rest_call = MethodType(
+                _rest_call_replacement, syn_client)
+            dataset_collections_to_consider_for_deletion = [f"'{ds}'" for ds in context[
+                "params"]["dataset_collections_for_cleanup"]]
+            query = f"SELECT * FROM {SYNAPSE_TABLE_FOR_CROISSANT_LINKS} WHERE dataset_collection in ({','.join(dataset_collections_to_consider_for_deletion)})"
+            table_results = syn_client.tableQuery(
+                query=query, resultsAs="csv")
+            table_dataframe = table_results.asDataFrame(
+                rowIdAndVersionInIndex=False)
+            os.remove(table_results.filepath)
+            rows_to_delete = extract_synapse_rows_to_delete(
+                synapse_rows=table_dataframe,
+                combined_dataset_collection_and_datasets=combined_dataset_collection_and_datasets,
+            )
+
+            if rows_to_delete:
+                delete_out_of_date_from_synapse = context["params"]["delete_out_of_date_from_synapse"]
+                if delete_out_of_date_from_synapse:
+                    otel_logger.info(
+                        f"Deleting the following rows from Synapse: {rows_to_delete}")
+                    results_from_synapse = syn_client.tableQuery(
+                        query=f"SELECT ROW_ID, ROW_VERSION FROM {SYNAPSE_TABLE_FOR_CROISSANT_LINKS} WHERE ROW_ID in ({','.join(rows_to_delete)})", resultsAs="csv")
+                    syn_client.delete(results_from_synapse)
+                    os.remove(results_from_synapse.filepath)
+                else:
+                    otel_logger.info(
+                        f"Found rows to delete from Synapse, but not deleting due to `delete_out_of_date_from_synapse` param: {rows_to_delete}")
+            else:
+                otel_logger.info(
+                    "No rows to delete from Synapse. All rows are current.")
+            otel_tracer.span_processor.force_flush()
+            otel_logger.handlers[0].flush()
+            return None
+
+    @task
+    def query_and_push_croissant_file(root_carrier_context: Dict, dataset_id: str, dataset_version: int, dataset_collection: str, **context) -> None:
         """
         Query the Snowflake database for the dataset in order to perform the Croissant
         transformation and push the results to S3.
 
         Arguments:
path: dags/synapse-dataset-to-croissant.py
user: login: linglp
body: since there are more arguments in the parameter, add more in the doc string? 
created_at: 2025-04-07T22:14:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -292,272 +867,277 @@ def combine_dataset_lists(root_carrier_context: Dict, dataset_ids: List[List[Dic
         Returns:
             The combined list of dataset IDs.
         """
-        with otel_tracer.start_as_current_span("combine_dataset_lists", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+        with otel_tracer.start_as_current_span("combine_dataset_lists", context=TraceContextTextMapPropagator().extract(root_carrier_context)):
             return_value = [
                 dataset for datasets in dataset_ids for dataset in datasets]
         otel_tracer.span_processor.force_flush()
         return return_value
 
     @task
-    def query_snowflake_and_push_croissant_file(root_carrier_context: Dict, dataset_id: str, **context) -> None:
+    def delete_non_current_croissant_file_in_s3(
+        root_carrier_context: Dict, combined_dataset_collection_and_datasets: List[Dict[str, str]], **context
+    ) -> None:
+        """
+        Delete the non-current croissant files from S3. 
+        This is used to remove the old files from S3 that are no longer needed. 
+        A "non-current" file is defined as a croissant JSON LD file which is no longer
+        present in any Dataset collection.
+
+        This can occur if the dataset has been removed from the dataset collection, or
+        if there is a new version of the dataset that has been added to the dataset
+        collection.
+
+        Arguments:
+            root_carrier_context: The root carrier context to use for the trace context.
+            combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+                dataset collections, and dataset versions to use to determine which
+                datasets to delete.
+
+        Returns:
+            None
+        """
+        with otel_tracer.start_as_current_span("delete_non_current_files_from_s3", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            s3_hook = S3Hook(
+                aws_conn_id=context["params"]["aws_conn_id"], region_name=REGION_NAME)
+            bucket_objects = s3_hook.list_keys(bucket_name=BUCKET_NAME)
+
+            objects_to_delete = extract_s3_objects_to_delete(
+                bucket_objects=bucket_objects,
+                dataset_collections_to_consider_for_deletion=context[
+                    "params"]["dataset_collections_for_cleanup"],
+                combined_dataset_collection_and_datasets=combined_dataset_collection_and_datasets,
+            )
+
+            if objects_to_delete:
+                delete_out_of_date_from_s3 = context["params"]["delete_out_of_date_from_s3"]
+                if delete_out_of_date_from_s3:
+                    otel_logger.info(
+                        f"Deleting the following objects from S3: {objects_to_delete}")
+                    s3_hook.delete_objects(
+                        bucket=BUCKET_NAME, keys=objects_to_delete)
+                else:
+                    otel_logger.info(
+                        f"Found objects to delete from S3, but not deleting due to `delete_out_of_date_from_s3` param: {objects_to_delete}")
+            else:
+                otel_logger.info(
+                    "No objects to delete from S3. All objects are current.")
+            otel_tracer.span_processor.force_flush()
+            otel_logger.handlers[0].flush()
+            return None
+
+    @task
+    def delete_non_current_croissant_file_in_synapse(
+        root_carrier_context: Dict, combined_dataset_collection_and_datasets: List[Dict[str, str]], **context
+    ) -> None:
+        """
+        Delete the non-current croissant files from the Synapse table. 
+        This is used to remove the old files from Synapse that are no longer needed. 
+        A "non-current" file is defined as a croissant JSON LD file which is no longer
+        present in any Dataset collection.
+
+        This can occur if the dataset has been removed from the dataset collection, or
+        if there is a new version of the dataset that has been added to the dataset
+        collection.
+
+        Arguments:
+            root_carrier_context: The root carrier context to use for the trace context.
+            combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+                dataset collections, and dataset versions to use to determine which
+                datasets to delete.
+
+        Returns:
+            None
+        """
+        # Warning: Using an authenticated Synapse Client during this section of code
+        with otel_tracer.start_as_current_span("delete_non_current_files_from_s3", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            syn_hook = SynapseHook(context["params"]["synapse_conn_id"])
+            syn_client: Synapse = syn_hook.client
+            syn_client._rest_call = MethodType(
+                _rest_call_replacement, syn_client)
+            dataset_collections_to_consider_for_deletion = [f"'{ds}'" for ds in context[
+                "params"]["dataset_collections_for_cleanup"]]
+            query = f"SELECT * FROM {SYNAPSE_TABLE_FOR_CROISSANT_LINKS} WHERE dataset_collection in ({','.join(dataset_collections_to_consider_for_deletion)})"
+            table_results = syn_client.tableQuery(
+                query=query, resultsAs="csv")
+            table_dataframe = table_results.asDataFrame(
+                rowIdAndVersionInIndex=False)
+            os.remove(table_results.filepath)
+            rows_to_delete = extract_synapse_rows_to_delete(
+                synapse_rows=table_dataframe,
+                combined_dataset_collection_and_datasets=combined_dataset_collection_and_datasets,
+            )
+
+            if rows_to_delete:
+                delete_out_of_date_from_synapse = context["params"]["delete_out_of_date_from_synapse"]
+                if delete_out_of_date_from_synapse:
+                    otel_logger.info(
+                        f"Deleting the following rows from Synapse: {rows_to_delete}")
+                    results_from_synapse = syn_client.tableQuery(
+                        query=f"SELECT ROW_ID, ROW_VERSION FROM {SYNAPSE_TABLE_FOR_CROISSANT_LINKS} WHERE ROW_ID in ({','.join(rows_to_delete)})", resultsAs="csv")
+                    syn_client.delete(results_from_synapse)
+                    os.remove(results_from_synapse.filepath)
+                else:
+                    otel_logger.info(
+                        f"Found rows to delete from Synapse, but not deleting due to `delete_out_of_date_from_synapse` param: {rows_to_delete}")
+            else:
+                otel_logger.info(
+                    "No rows to delete from Synapse. All rows are current.")
+            otel_tracer.span_processor.force_flush()
+            otel_logger.handlers[0].flush()
+            return None
+
+    @task
+    def query_and_push_croissant_file(root_carrier_context: Dict, dataset_id: str, dataset_version: int, dataset_collection: str, **context) -> None:
         """
         Query the Snowflake database for the dataset in order to perform the Croissant
         transformation and push the results to S3.
 
         Arguments:
             dataset_id: The ID of the dataset to query in Snowflake.
         """
-        with otel_tracer.start_as_current_span("query_snowflake_and_push_croissant_file", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+        with otel_tracer.start_as_current_span("query_and_push_croissant_file", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            push_to_s3 = context["params"]["push_results_to_s3"]
+            push_to_synapse = context["params"]["push_links_to_synapse"]
+            span.set_attribute("airflow.push_results_to_s3", push_to_s3)
+            span.set_attribute(
+                "airflow.push_links_to_synapse", push_to_synapse)
             if not dataset_id:
                 otel_logger.info("No dataset found")
                 otel_tracer.span_processor.force_flush()
                 otel_logger.handlers[0].flush()
                 return
 
-            span.set_attribute("airflow.dataset_id", f"syn{dataset_id}")
-
-            query = f"""
-            WITH entity_metadata AS (
-                SELECT
-                    items,
-                    name,
-                    ANNOTATIONS,
-                    CREATED_ON,
-                    VERSION_NUMBER,
-                    CONCAT('syn', id) as DATASET_SYNAPSE_ID
-                FROM synapse_data_warehouse.synapse.node_latest
-                WHERE id = {dataset_id}
-            ),
-            ids_of_files_belonging_to_dataset AS (
-                SELECT REPLACE(ITEM.value:entityId::STRING, 'syn', '') AS entity_id, ITEM.value:versionNumber::STRING as version_number
-                FROM entity_metadata,
-                LATERAL FLATTEN(input => entity_metadata.ITEMS) ITEM
-            ),
-            files_belonging_to_dataset AS (
-                SELECT
-                    nl.id,
-                    nl.version_number,
-                    vh.value AS VERSION_HISTORY_ENTRY,
-                    nl.ANNOTATIONS,
-                    nl.is_public,
-                    ids_of_files_belonging_to_dataset.entity_id,
-                    ids_of_files_belonging_to_dataset.version_number,
-                    entity_metadata.DATASET_SYNAPSE_ID
-                FROM synapse_data_warehouse.synapse.node_latest AS nl
-                JOIN ids_of_files_belonging_to_dataset
-                    ON nl.id = ids_of_files_belonging_to_dataset.entity_id
-                JOIN entity_metadata
-                    ON TRUE -- Ensures the `DATASET_SYNAPSE_ID` is included
-                CROSS JOIN LATERAL FLATTEN(input => nl.VERSION_HISTORY) vh
-                WHERE nl.is_public = TRUE
-                AND vh.value:versionNumber::STRING = ids_of_files_belonging_to_dataset.version_number
-            ),
-            distribution_pointing_to_files AS (
-                SELECT
-                    CONCAT('syn', id) as SYNAPSE_ID,
-                    DATASET_SYNAPSE_ID,
-                    OBJECT_CONSTRUCT(
-                        '@type', 'FileObject',
-                        '@id', CONCAT('syn', id),
-                        'name', CONCAT('syn', id),
-                        'description', CONCAT(
-                            'Data file associated with ', CONCAT('syn', id)),
-                        'contentUrl', CONCAT(
-                            'https://www.synapse.org/Synapse:', CONCAT('syn', id)),
-                        'encodingFormat', 'application/json',
-                        'md5', COALESCE(VERSION_HISTORY_ENTRY:contentMd5, 'unknown_md5'),
-                        'sha256', 'unknown'
-                    ) AS file_object
-                FROM files_belonging_to_dataset
-            ),
-            -- TODO: Annotation selection must be version dependent, however, the different versions of the annotations are not available in a non-deduplicated table
-            distinct_annotation_keys AS (
-                SELECT DATASET_SYNAPSE_ID, ARRAY_AGG(annotation_key) as annotation_keys FROM (
-                    SELECT
-                        DATASET_SYNAPSE_ID,
-                        key as annotation_key
-                    FROM files_belonging_to_dataset,
-                    LATERAL FLATTEN(input => ANNOTATIONS:annotations) AS annotation
-                    GROUP BY DATASET_SYNAPSE_ID, key
-                ) GROUP BY DATASET_SYNAPSE_ID
-            ),
-            fields_pointing_to_metadata AS (
-                SELECT
-                    DATASET_SYNAPSE_ID,
-                    OBJECT_CONSTRUCT(
-                        '@type', 'RecordSet',
-                        '@id', 'default',
-                        'name', 'default',
-                        'description', 'Metadata for the dataset',
-                        'field', ARRAY_AGG(
-                            OBJECT_CONSTRUCT(
-                                '@type', 'cr:Field',
-                                '@id', CONCAT('metadata', '/', value),
-                                'name', value,
-                                -- 'description', '',
-                                'dataType', 'sc:Text',
-                                'source', OBJECT_CONSTRUCT(
-                                    'fileObject', OBJECT_CONSTRUCT(
-                                        '@id', 'metadata'),
-                                    'extract', OBJECT_CONSTRUCT('column', value)
-                                )
-                            )
-                        )
-                    ) AS record_set
-                FROM distinct_annotation_keys,
-                LATERAL FLATTEN(input => distinct_annotation_keys.annotation_keys)
-                GROUP BY DATASET_SYNAPSE_ID
-            ),
-            croissant_metadata AS (
-                SELECT
-                    entity_metadata.DATASET_SYNAPSE_ID,
-                    entity_metadata.name,
-                    OBJECT_CONSTRUCT(
-                        '@context', OBJECT_CONSTRUCT(
-                            '@language', 'en',
-                            '@vocab', 'https://schema.org/',
-                            'citeAs', 'cr:citeAs',
-                            'column', 'cr:column',
-                            'conformsTo', 'dct:conformsTo',
-                            'cr', 'http://mlcommons.org/croissant/',
-                            'rai', 'http://mlcommons.org/croissant/RAI/',
-                            'data', OBJECT_CONSTRUCT(
-                                '@id', 'cr:data', '@type', '@json'),
-                            'dataType', OBJECT_CONSTRUCT(
-                                '@id', 'cr:dataType', '@type', '@vocab'),
-                            'dct', 'http://purl.org/dc/terms/',
-                            'examples', OBJECT_CONSTRUCT(
-                                '@id', 'cr:examples', '@type', '@json'),
-                            'extract', 'cr:extract',
-                            'field', 'cr:field',
-                            'fileProperty', 'cr:fileProperty',
-                            'fileObject', 'cr:fileObject',
-                            'fileSet', 'cr:fileSet',
-                            'format', 'cr:format',
-                            'includes', 'cr:includes',
-                            'isLiveDataset', 'cr:isLiveDataset',
-                            'jsonPath', 'cr:jsonPath',
-                            'key', 'cr:key',
-                            'md5', 'cr:md5',
-                            'parentField', 'cr:parentField',
-                            'path', 'cr:path',
-                            'recordSet', 'cr:recordSet',
-                            'references', 'cr:references',
-                            'regex', 'cr:regex',
-                            'repeated', 'cr:repeated',
-                            'replace', 'cr:replace',
-                            'sc', 'https://schema.org/',
-                            'separator', 'cr:separator',
-                            'source', 'cr:source',
-                            'subField', 'cr:subField',
-                            'transform', 'cr:transform'
-                        ),
-                        '@type', 'Dataset',
-                        'name', COALESCE(entity_metadata.name, 'Synapse Dataset'),
-                        'description', COALESCE(ANNOTATIONS:annotations:description:value[0], 'A dataset containing annotations for genomic data from Synapse.'),
-                        'url', CONCAT('https://www.synapse.org/Synapse:',
-                                    entity_metadata.DATASET_SYNAPSE_ID),
-                        'citation', COALESCE(ANNOTATIONS:annotations:citation:value[0], 'unknown_citation'),
-                        'datePublished', COALESCE(CREATED_ON, 'unknown_date'),
-                        'license', COALESCE(ANNOTATIONS:annotations:license:value[0], 'unknown_license'),
-                        'version', COALESCE(VERSION_NUMBER, 'unknown_version'),
-                        'distribution', ARRAY_PREPEND(ARRAY_AGG(distribution_pointing_to_files.file_object),
-                                            OBJECT_CONSTRUCT(
-                                                '@type', 'FileObject',
-                                                '@id', 'metadata',
-                                                'name', 'metadata',
-                                                'description', CONCAT(
-                                                    'Metadata associated with ', entity_metadata.DATASET_SYNAPSE_ID),
-                                                'contentUrl', CONCAT(
-                                                    'https://www.synapse.org/Synapse:', entity_metadata.DATASET_SYNAPSE_ID),
-                                                'encodingFormat', 'application/csv',
-                                                'md5', '',
-                                                'sha256', 'unknown'
-                                            )
-                                        ),
-                        'recordSet', fields_pointing_to_metadata.record_set
-                    ) AS metadata
-                FROM
-                    entity_metadata, fields_pointing_to_metadata
-                JOIN
-                    distribution_pointing_to_files
-                ON
-                    distribution_pointing_to_files.DATASET_SYNAPSE_ID = fields_pointing_to_metadata.DATASET_SYNAPSE_ID
-                GROUP BY
-                    entity_metadata.DATASET_SYNAPSE_ID,
-                    entity_metadata.name,
-                    fields_pointing_to_metadata.DATASET_SYNAPSE_ID,
-                    fields_pointing_to_metadata.record_set,
-                    entity_metadata.name,
-                    entity_metadata.ANNOTATIONS,
-                    entity_metadata.CREATED_ON,
-                    entity_metadata.VERSION_NUMBER
-            )
-
-            SELECT
-                metadata::STRING AS croissant_metadata,
-                DATASET_SYNAPSE_ID,
-                name
-            FROM
-                croissant_metadata;
-            """
-            snow_hook = SnowflakeHook(context["params"]["snowflake_developer_service_conn"])
-            ctx = snow_hook.get_conn()
-            cs = ctx.cursor()
-
-            otel_logger.info(
-                f"Performing query for dataset {dataset_id}")
+            span.set_attribute("airflow.dataset_id", f"{dataset_id}")
+            path_to_remove = None
+            syn_client = create_syn_client()
+            file_ids_and_versions_attached_to_dataset = []
 
             try:
path: dags/synapse-dataset-to-croissant.py
user: login: linglp
body: starting from this try and except block, could you add more comments in the code?
created_at: 2025-04-07T23:02:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -292,272 +867,277 @@ def combine_dataset_lists(root_carrier_context: Dict, dataset_ids: List[List[Dic
         Returns:
             The combined list of dataset IDs.
         """
-        with otel_tracer.start_as_current_span("combine_dataset_lists", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+        with otel_tracer.start_as_current_span("combine_dataset_lists", context=TraceContextTextMapPropagator().extract(root_carrier_context)):
             return_value = [
                 dataset for datasets in dataset_ids for dataset in datasets]
         otel_tracer.span_processor.force_flush()
         return return_value
 
     @task
-    def query_snowflake_and_push_croissant_file(root_carrier_context: Dict, dataset_id: str, **context) -> None:
+    def delete_non_current_croissant_file_in_s3(
+        root_carrier_context: Dict, combined_dataset_collection_and_datasets: List[Dict[str, str]], **context
+    ) -> None:
+        """
+        Delete the non-current croissant files from S3. 
+        This is used to remove the old files from S3 that are no longer needed. 
+        A "non-current" file is defined as a croissant JSON LD file which is no longer
+        present in any Dataset collection.
+
+        This can occur if the dataset has been removed from the dataset collection, or
+        if there is a new version of the dataset that has been added to the dataset
+        collection.
+
+        Arguments:
+            root_carrier_context: The root carrier context to use for the trace context.
+            combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+                dataset collections, and dataset versions to use to determine which
+                datasets to delete.
+
+        Returns:
+            None
+        """
+        with otel_tracer.start_as_current_span("delete_non_current_files_from_s3", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            s3_hook = S3Hook(
+                aws_conn_id=context["params"]["aws_conn_id"], region_name=REGION_NAME)
+            bucket_objects = s3_hook.list_keys(bucket_name=BUCKET_NAME)
+
+            objects_to_delete = extract_s3_objects_to_delete(
+                bucket_objects=bucket_objects,
+                dataset_collections_to_consider_for_deletion=context[
+                    "params"]["dataset_collections_for_cleanup"],
+                combined_dataset_collection_and_datasets=combined_dataset_collection_and_datasets,
+            )
+
+            if objects_to_delete:
+                delete_out_of_date_from_s3 = context["params"]["delete_out_of_date_from_s3"]
+                if delete_out_of_date_from_s3:
+                    otel_logger.info(
+                        f"Deleting the following objects from S3: {objects_to_delete}")
+                    s3_hook.delete_objects(
+                        bucket=BUCKET_NAME, keys=objects_to_delete)
+                else:
+                    otel_logger.info(
path: dags/synapse-dataset-to-croissant.py
user: login: BryanFauble
body: There is a boolean `delete_out_of_date_from_s3` that can be toggled when you run the DAG. When this is false we will be running this portion of the code in "dry run" mode. So it'll print what it would do, without doing it.
created_at: 2025-04-07T23:16:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -292,272 +867,277 @@ def combine_dataset_lists(root_carrier_context: Dict, dataset_ids: List[List[Dic
         Returns:
             The combined list of dataset IDs.
         """
-        with otel_tracer.start_as_current_span("combine_dataset_lists", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+        with otel_tracer.start_as_current_span("combine_dataset_lists", context=TraceContextTextMapPropagator().extract(root_carrier_context)):
             return_value = [
                 dataset for datasets in dataset_ids for dataset in datasets]
         otel_tracer.span_processor.force_flush()
         return return_value
 
     @task
-    def query_snowflake_and_push_croissant_file(root_carrier_context: Dict, dataset_id: str, **context) -> None:
+    def delete_non_current_croissant_file_in_s3(
+        root_carrier_context: Dict, combined_dataset_collection_and_datasets: List[Dict[str, str]], **context
+    ) -> None:
+        """
+        Delete the non-current croissant files from S3. 
+        This is used to remove the old files from S3 that are no longer needed. 
+        A "non-current" file is defined as a croissant JSON LD file which is no longer
+        present in any Dataset collection.
+
+        This can occur if the dataset has been removed from the dataset collection, or
+        if there is a new version of the dataset that has been added to the dataset
+        collection.
+
+        Arguments:
+            root_carrier_context: The root carrier context to use for the trace context.
+            combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+                dataset collections, and dataset versions to use to determine which
+                datasets to delete.
+
+        Returns:
+            None
+        """
+        with otel_tracer.start_as_current_span("delete_non_current_files_from_s3", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            s3_hook = S3Hook(
+                aws_conn_id=context["params"]["aws_conn_id"], region_name=REGION_NAME)
+            bucket_objects = s3_hook.list_keys(bucket_name=BUCKET_NAME)
+
+            objects_to_delete = extract_s3_objects_to_delete(
+                bucket_objects=bucket_objects,
+                dataset_collections_to_consider_for_deletion=context[
+                    "params"]["dataset_collections_for_cleanup"],
+                combined_dataset_collection_and_datasets=combined_dataset_collection_and_datasets,
+            )
+
+            if objects_to_delete:
+                delete_out_of_date_from_s3 = context["params"]["delete_out_of_date_from_s3"]
+                if delete_out_of_date_from_s3:
+                    otel_logger.info(
+                        f"Deleting the following objects from S3: {objects_to_delete}")
+                    s3_hook.delete_objects(
+                        bucket=BUCKET_NAME, keys=objects_to_delete)
+                else:
+                    otel_logger.info(
+                        f"Found objects to delete from S3, but not deleting due to `delete_out_of_date_from_s3` param: {objects_to_delete}")
+            else:
+                otel_logger.info(
+                    "No objects to delete from S3. All objects are current.")
+            otel_tracer.span_processor.force_flush()
+            otel_logger.handlers[0].flush()
+            return None
+
+    @task
+    def delete_non_current_croissant_file_in_synapse(
+        root_carrier_context: Dict, combined_dataset_collection_and_datasets: List[Dict[str, str]], **context
+    ) -> None:
+        """
+        Delete the non-current croissant files from the Synapse table. 
+        This is used to remove the old files from Synapse that are no longer needed. 
+        A "non-current" file is defined as a croissant JSON LD file which is no longer
+        present in any Dataset collection.
+
+        This can occur if the dataset has been removed from the dataset collection, or
+        if there is a new version of the dataset that has been added to the dataset
+        collection.
+
+        Arguments:
+            root_carrier_context: The root carrier context to use for the trace context.
+            combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+                dataset collections, and dataset versions to use to determine which
+                datasets to delete.
+
+        Returns:
+            None
+        """
+        # Warning: Using an authenticated Synapse Client during this section of code
+        with otel_tracer.start_as_current_span("delete_non_current_files_from_s3", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            syn_hook = SynapseHook(context["params"]["synapse_conn_id"])
+            syn_client: Synapse = syn_hook.client
+            syn_client._rest_call = MethodType(
+                _rest_call_replacement, syn_client)
path: dags/synapse-dataset-to-croissant.py
user: login: BryanFauble
body: At the moment this isn't much of a concern IMO. It'll be fixed in the next release of the SYNPY client and we can remove this hacky code at that point
created_at: 2025-04-07T23:17:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -292,272 +867,277 @@ def combine_dataset_lists(root_carrier_context: Dict, dataset_ids: List[List[Dic
         Returns:
             The combined list of dataset IDs.
         """
-        with otel_tracer.start_as_current_span("combine_dataset_lists", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+        with otel_tracer.start_as_current_span("combine_dataset_lists", context=TraceContextTextMapPropagator().extract(root_carrier_context)):
             return_value = [
                 dataset for datasets in dataset_ids for dataset in datasets]
         otel_tracer.span_processor.force_flush()
         return return_value
 
     @task
-    def query_snowflake_and_push_croissant_file(root_carrier_context: Dict, dataset_id: str, **context) -> None:
+    def delete_non_current_croissant_file_in_s3(
+        root_carrier_context: Dict, combined_dataset_collection_and_datasets: List[Dict[str, str]], **context
+    ) -> None:
+        """
+        Delete the non-current croissant files from S3. 
+        This is used to remove the old files from S3 that are no longer needed. 
+        A "non-current" file is defined as a croissant JSON LD file which is no longer
+        present in any Dataset collection.
+
+        This can occur if the dataset has been removed from the dataset collection, or
+        if there is a new version of the dataset that has been added to the dataset
+        collection.
+
+        Arguments:
+            root_carrier_context: The root carrier context to use for the trace context.
+            combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+                dataset collections, and dataset versions to use to determine which
+                datasets to delete.
+
+        Returns:
+            None
+        """
+        with otel_tracer.start_as_current_span("delete_non_current_files_from_s3", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            s3_hook = S3Hook(
+                aws_conn_id=context["params"]["aws_conn_id"], region_name=REGION_NAME)
+            bucket_objects = s3_hook.list_keys(bucket_name=BUCKET_NAME)
+
+            objects_to_delete = extract_s3_objects_to_delete(
+                bucket_objects=bucket_objects,
+                dataset_collections_to_consider_for_deletion=context[
+                    "params"]["dataset_collections_for_cleanup"],
+                combined_dataset_collection_and_datasets=combined_dataset_collection_and_datasets,
+            )
+
+            if objects_to_delete:
+                delete_out_of_date_from_s3 = context["params"]["delete_out_of_date_from_s3"]
+                if delete_out_of_date_from_s3:
+                    otel_logger.info(
+                        f"Deleting the following objects from S3: {objects_to_delete}")
+                    s3_hook.delete_objects(
+                        bucket=BUCKET_NAME, keys=objects_to_delete)
+                else:
+                    otel_logger.info(
+                        f"Found objects to delete from S3, but not deleting due to `delete_out_of_date_from_s3` param: {objects_to_delete}")
+            else:
+                otel_logger.info(
+                    "No objects to delete from S3. All objects are current.")
+            otel_tracer.span_processor.force_flush()
+            otel_logger.handlers[0].flush()
+            return None
+
+    @task
+    def delete_non_current_croissant_file_in_synapse(
+        root_carrier_context: Dict, combined_dataset_collection_and_datasets: List[Dict[str, str]], **context
+    ) -> None:
+        """
+        Delete the non-current croissant files from the Synapse table. 
+        This is used to remove the old files from Synapse that are no longer needed. 
+        A "non-current" file is defined as a croissant JSON LD file which is no longer
+        present in any Dataset collection.
+
+        This can occur if the dataset has been removed from the dataset collection, or
+        if there is a new version of the dataset that has been added to the dataset
+        collection.
+
+        Arguments:
+            root_carrier_context: The root carrier context to use for the trace context.
+            combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+                dataset collections, and dataset versions to use to determine which
+                datasets to delete.
+
+        Returns:
+            None
+        """
+        # Warning: Using an authenticated Synapse Client during this section of code
+        with otel_tracer.start_as_current_span("delete_non_current_files_from_s3", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            syn_hook = SynapseHook(context["params"]["synapse_conn_id"])
+            syn_client: Synapse = syn_hook.client
+            syn_client._rest_call = MethodType(
+                _rest_call_replacement, syn_client)
+            dataset_collections_to_consider_for_deletion = [f"'{ds}'" for ds in context[
+                "params"]["dataset_collections_for_cleanup"]]
+            query = f"SELECT * FROM {SYNAPSE_TABLE_FOR_CROISSANT_LINKS} WHERE dataset_collection in ({','.join(dataset_collections_to_consider_for_deletion)})"
+            table_results = syn_client.tableQuery(
+                query=query, resultsAs="csv")
+            table_dataframe = table_results.asDataFrame(
+                rowIdAndVersionInIndex=False)
+            os.remove(table_results.filepath)
+            rows_to_delete = extract_synapse_rows_to_delete(
+                synapse_rows=table_dataframe,
+                combined_dataset_collection_and_datasets=combined_dataset_collection_and_datasets,
+            )
+
+            if rows_to_delete:
+                delete_out_of_date_from_synapse = context["params"]["delete_out_of_date_from_synapse"]
+                if delete_out_of_date_from_synapse:
+                    otel_logger.info(
+                        f"Deleting the following rows from Synapse: {rows_to_delete}")
+                    results_from_synapse = syn_client.tableQuery(
+                        query=f"SELECT ROW_ID, ROW_VERSION FROM {SYNAPSE_TABLE_FOR_CROISSANT_LINKS} WHERE ROW_ID in ({','.join(rows_to_delete)})", resultsAs="csv")
+                    syn_client.delete(results_from_synapse)
+                    os.remove(results_from_synapse.filepath)
+                else:
+                    otel_logger.info(
+                        f"Found rows to delete from Synapse, but not deleting due to `delete_out_of_date_from_synapse` param: {rows_to_delete}")
+            else:
+                otel_logger.info(
+                    "No rows to delete from Synapse. All rows are current.")
+            otel_tracer.span_processor.force_flush()
+            otel_logger.handlers[0].flush()
+            return None
+
+    @task
+    def query_and_push_croissant_file(root_carrier_context: Dict, dataset_id: str, dataset_version: int, dataset_collection: str, **context) -> None:
         """
         Query the Snowflake database for the dataset in order to perform the Croissant
         transformation and push the results to S3.
 
         Arguments:
             dataset_id: The ID of the dataset to query in Snowflake.
         """
-        with otel_tracer.start_as_current_span("query_snowflake_and_push_croissant_file", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+        with otel_tracer.start_as_current_span("query_and_push_croissant_file", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            push_to_s3 = context["params"]["push_results_to_s3"]
+            push_to_synapse = context["params"]["push_links_to_synapse"]
+            span.set_attribute("airflow.push_results_to_s3", push_to_s3)
+            span.set_attribute(
+                "airflow.push_links_to_synapse", push_to_synapse)
             if not dataset_id:
                 otel_logger.info("No dataset found")
                 otel_tracer.span_processor.force_flush()
                 otel_logger.handlers[0].flush()
                 return
 
-            span.set_attribute("airflow.dataset_id", f"syn{dataset_id}")
-
-            query = f"""
-            WITH entity_metadata AS (
-                SELECT
-                    items,
-                    name,
-                    ANNOTATIONS,
-                    CREATED_ON,
-                    VERSION_NUMBER,
-                    CONCAT('syn', id) as DATASET_SYNAPSE_ID
-                FROM synapse_data_warehouse.synapse.node_latest
-                WHERE id = {dataset_id}
-            ),
-            ids_of_files_belonging_to_dataset AS (
-                SELECT REPLACE(ITEM.value:entityId::STRING, 'syn', '') AS entity_id, ITEM.value:versionNumber::STRING as version_number
-                FROM entity_metadata,
-                LATERAL FLATTEN(input => entity_metadata.ITEMS) ITEM
-            ),
-            files_belonging_to_dataset AS (
-                SELECT
-                    nl.id,
-                    nl.version_number,
-                    vh.value AS VERSION_HISTORY_ENTRY,
-                    nl.ANNOTATIONS,
-                    nl.is_public,
-                    ids_of_files_belonging_to_dataset.entity_id,
-                    ids_of_files_belonging_to_dataset.version_number,
-                    entity_metadata.DATASET_SYNAPSE_ID
-                FROM synapse_data_warehouse.synapse.node_latest AS nl
-                JOIN ids_of_files_belonging_to_dataset
-                    ON nl.id = ids_of_files_belonging_to_dataset.entity_id
-                JOIN entity_metadata
-                    ON TRUE -- Ensures the `DATASET_SYNAPSE_ID` is included
-                CROSS JOIN LATERAL FLATTEN(input => nl.VERSION_HISTORY) vh
-                WHERE nl.is_public = TRUE
-                AND vh.value:versionNumber::STRING = ids_of_files_belonging_to_dataset.version_number
-            ),
-            distribution_pointing_to_files AS (
-                SELECT
-                    CONCAT('syn', id) as SYNAPSE_ID,
-                    DATASET_SYNAPSE_ID,
-                    OBJECT_CONSTRUCT(
-                        '@type', 'FileObject',
-                        '@id', CONCAT('syn', id),
-                        'name', CONCAT('syn', id),
-                        'description', CONCAT(
-                            'Data file associated with ', CONCAT('syn', id)),
-                        'contentUrl', CONCAT(
-                            'https://www.synapse.org/Synapse:', CONCAT('syn', id)),
-                        'encodingFormat', 'application/json',
-                        'md5', COALESCE(VERSION_HISTORY_ENTRY:contentMd5, 'unknown_md5'),
-                        'sha256', 'unknown'
-                    ) AS file_object
-                FROM files_belonging_to_dataset
-            ),
-            -- TODO: Annotation selection must be version dependent, however, the different versions of the annotations are not available in a non-deduplicated table
-            distinct_annotation_keys AS (
-                SELECT DATASET_SYNAPSE_ID, ARRAY_AGG(annotation_key) as annotation_keys FROM (
-                    SELECT
-                        DATASET_SYNAPSE_ID,
-                        key as annotation_key
-                    FROM files_belonging_to_dataset,
-                    LATERAL FLATTEN(input => ANNOTATIONS:annotations) AS annotation
-                    GROUP BY DATASET_SYNAPSE_ID, key
-                ) GROUP BY DATASET_SYNAPSE_ID
-            ),
-            fields_pointing_to_metadata AS (
-                SELECT
-                    DATASET_SYNAPSE_ID,
-                    OBJECT_CONSTRUCT(
-                        '@type', 'RecordSet',
-                        '@id', 'default',
-                        'name', 'default',
-                        'description', 'Metadata for the dataset',
-                        'field', ARRAY_AGG(
-                            OBJECT_CONSTRUCT(
-                                '@type', 'cr:Field',
-                                '@id', CONCAT('metadata', '/', value),
-                                'name', value,
-                                -- 'description', '',
-                                'dataType', 'sc:Text',
-                                'source', OBJECT_CONSTRUCT(
-                                    'fileObject', OBJECT_CONSTRUCT(
-                                        '@id', 'metadata'),
-                                    'extract', OBJECT_CONSTRUCT('column', value)
-                                )
-                            )
-                        )
-                    ) AS record_set
-                FROM distinct_annotation_keys,
-                LATERAL FLATTEN(input => distinct_annotation_keys.annotation_keys)
-                GROUP BY DATASET_SYNAPSE_ID
-            ),
-            croissant_metadata AS (
-                SELECT
-                    entity_metadata.DATASET_SYNAPSE_ID,
-                    entity_metadata.name,
-                    OBJECT_CONSTRUCT(
-                        '@context', OBJECT_CONSTRUCT(
-                            '@language', 'en',
-                            '@vocab', 'https://schema.org/',
-                            'citeAs', 'cr:citeAs',
-                            'column', 'cr:column',
-                            'conformsTo', 'dct:conformsTo',
-                            'cr', 'http://mlcommons.org/croissant/',
-                            'rai', 'http://mlcommons.org/croissant/RAI/',
-                            'data', OBJECT_CONSTRUCT(
-                                '@id', 'cr:data', '@type', '@json'),
-                            'dataType', OBJECT_CONSTRUCT(
-                                '@id', 'cr:dataType', '@type', '@vocab'),
-                            'dct', 'http://purl.org/dc/terms/',
-                            'examples', OBJECT_CONSTRUCT(
-                                '@id', 'cr:examples', '@type', '@json'),
-                            'extract', 'cr:extract',
-                            'field', 'cr:field',
-                            'fileProperty', 'cr:fileProperty',
-                            'fileObject', 'cr:fileObject',
-                            'fileSet', 'cr:fileSet',
-                            'format', 'cr:format',
-                            'includes', 'cr:includes',
-                            'isLiveDataset', 'cr:isLiveDataset',
-                            'jsonPath', 'cr:jsonPath',
-                            'key', 'cr:key',
-                            'md5', 'cr:md5',
-                            'parentField', 'cr:parentField',
-                            'path', 'cr:path',
-                            'recordSet', 'cr:recordSet',
-                            'references', 'cr:references',
-                            'regex', 'cr:regex',
-                            'repeated', 'cr:repeated',
-                            'replace', 'cr:replace',
-                            'sc', 'https://schema.org/',
-                            'separator', 'cr:separator',
-                            'source', 'cr:source',
-                            'subField', 'cr:subField',
-                            'transform', 'cr:transform'
-                        ),
-                        '@type', 'Dataset',
-                        'name', COALESCE(entity_metadata.name, 'Synapse Dataset'),
-                        'description', COALESCE(ANNOTATIONS:annotations:description:value[0], 'A dataset containing annotations for genomic data from Synapse.'),
-                        'url', CONCAT('https://www.synapse.org/Synapse:',
-                                    entity_metadata.DATASET_SYNAPSE_ID),
-                        'citation', COALESCE(ANNOTATIONS:annotations:citation:value[0], 'unknown_citation'),
-                        'datePublished', COALESCE(CREATED_ON, 'unknown_date'),
-                        'license', COALESCE(ANNOTATIONS:annotations:license:value[0], 'unknown_license'),
-                        'version', COALESCE(VERSION_NUMBER, 'unknown_version'),
-                        'distribution', ARRAY_PREPEND(ARRAY_AGG(distribution_pointing_to_files.file_object),
-                                            OBJECT_CONSTRUCT(
-                                                '@type', 'FileObject',
-                                                '@id', 'metadata',
-                                                'name', 'metadata',
-                                                'description', CONCAT(
-                                                    'Metadata associated with ', entity_metadata.DATASET_SYNAPSE_ID),
-                                                'contentUrl', CONCAT(
-                                                    'https://www.synapse.org/Synapse:', entity_metadata.DATASET_SYNAPSE_ID),
-                                                'encodingFormat', 'application/csv',
-                                                'md5', '',
-                                                'sha256', 'unknown'
-                                            )
-                                        ),
-                        'recordSet', fields_pointing_to_metadata.record_set
-                    ) AS metadata
-                FROM
-                    entity_metadata, fields_pointing_to_metadata
-                JOIN
-                    distribution_pointing_to_files
-                ON
-                    distribution_pointing_to_files.DATASET_SYNAPSE_ID = fields_pointing_to_metadata.DATASET_SYNAPSE_ID
-                GROUP BY
-                    entity_metadata.DATASET_SYNAPSE_ID,
-                    entity_metadata.name,
-                    fields_pointing_to_metadata.DATASET_SYNAPSE_ID,
-                    fields_pointing_to_metadata.record_set,
-                    entity_metadata.name,
-                    entity_metadata.ANNOTATIONS,
-                    entity_metadata.CREATED_ON,
-                    entity_metadata.VERSION_NUMBER
-            )
-
-            SELECT
-                metadata::STRING AS croissant_metadata,
-                DATASET_SYNAPSE_ID,
-                name
-            FROM
-                croissant_metadata;
-            """
-            snow_hook = SnowflakeHook(context["params"]["snowflake_developer_service_conn"])
-            ctx = snow_hook.get_conn()
-            cs = ctx.cursor()
-
-            otel_logger.info(
-                f"Performing query for dataset {dataset_id}")
+            span.set_attribute("airflow.dataset_id", f"{dataset_id}")
+            path_to_remove = None
+            syn_client = create_syn_client()
+            file_ids_and_versions_attached_to_dataset = []
 
             try:
path: dags/synapse-dataset-to-croissant.py
user: login: BryanFauble
body: I added a few more! [Take a look at this commit](https://github.com/Sage-Bionetworks-Workflows/orca-recipes/pull/83/commits/616a6e1d8cb81aa69762f59cffc6d452a0591f91)
created_at: 2025-04-07T23:25:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -292,272 +867,277 @@ def combine_dataset_lists(root_carrier_context: Dict, dataset_ids: List[List[Dic
         Returns:
             The combined list of dataset IDs.
         """
-        with otel_tracer.start_as_current_span("combine_dataset_lists", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+        with otel_tracer.start_as_current_span("combine_dataset_lists", context=TraceContextTextMapPropagator().extract(root_carrier_context)):
             return_value = [
                 dataset for datasets in dataset_ids for dataset in datasets]
         otel_tracer.span_processor.force_flush()
         return return_value
 
     @task
-    def query_snowflake_and_push_croissant_file(root_carrier_context: Dict, dataset_id: str, **context) -> None:
+    def delete_non_current_croissant_file_in_s3(
+        root_carrier_context: Dict, combined_dataset_collection_and_datasets: List[Dict[str, str]], **context
+    ) -> None:
+        """
+        Delete the non-current croissant files from S3. 
+        This is used to remove the old files from S3 that are no longer needed. 
+        A "non-current" file is defined as a croissant JSON LD file which is no longer
+        present in any Dataset collection.
+
+        This can occur if the dataset has been removed from the dataset collection, or
+        if there is a new version of the dataset that has been added to the dataset
+        collection.
+
+        Arguments:
+            root_carrier_context: The root carrier context to use for the trace context.
+            combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+                dataset collections, and dataset versions to use to determine which
+                datasets to delete.
+
+        Returns:
+            None
+        """
+        with otel_tracer.start_as_current_span("delete_non_current_files_from_s3", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            s3_hook = S3Hook(
+                aws_conn_id=context["params"]["aws_conn_id"], region_name=REGION_NAME)
+            bucket_objects = s3_hook.list_keys(bucket_name=BUCKET_NAME)
+
+            objects_to_delete = extract_s3_objects_to_delete(
+                bucket_objects=bucket_objects,
+                dataset_collections_to_consider_for_deletion=context[
+                    "params"]["dataset_collections_for_cleanup"],
+                combined_dataset_collection_and_datasets=combined_dataset_collection_and_datasets,
+            )
+
+            if objects_to_delete:
+                delete_out_of_date_from_s3 = context["params"]["delete_out_of_date_from_s3"]
+                if delete_out_of_date_from_s3:
+                    otel_logger.info(
+                        f"Deleting the following objects from S3: {objects_to_delete}")
+                    s3_hook.delete_objects(
+                        bucket=BUCKET_NAME, keys=objects_to_delete)
+                else:
+                    otel_logger.info(
+                        f"Found objects to delete from S3, but not deleting due to `delete_out_of_date_from_s3` param: {objects_to_delete}")
+            else:
+                otel_logger.info(
+                    "No objects to delete from S3. All objects are current.")
+            otel_tracer.span_processor.force_flush()
+            otel_logger.handlers[0].flush()
+            return None
+
+    @task
+    def delete_non_current_croissant_file_in_synapse(
+        root_carrier_context: Dict, combined_dataset_collection_and_datasets: List[Dict[str, str]], **context
+    ) -> None:
+        """
+        Delete the non-current croissant files from the Synapse table. 
+        This is used to remove the old files from Synapse that are no longer needed. 
+        A "non-current" file is defined as a croissant JSON LD file which is no longer
+        present in any Dataset collection.
+
+        This can occur if the dataset has been removed from the dataset collection, or
+        if there is a new version of the dataset that has been added to the dataset
+        collection.
+
+        Arguments:
+            root_carrier_context: The root carrier context to use for the trace context.
+            combined_dataset_collection_and_datasets: The combined list of dataset IDs,
+                dataset collections, and dataset versions to use to determine which
+                datasets to delete.
+
+        Returns:
+            None
+        """
+        # Warning: Using an authenticated Synapse Client during this section of code
+        with otel_tracer.start_as_current_span("delete_non_current_files_from_s3", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            syn_hook = SynapseHook(context["params"]["synapse_conn_id"])
+            syn_client: Synapse = syn_hook.client
+            syn_client._rest_call = MethodType(
+                _rest_call_replacement, syn_client)
+            dataset_collections_to_consider_for_deletion = [f"'{ds}'" for ds in context[
+                "params"]["dataset_collections_for_cleanup"]]
+            query = f"SELECT * FROM {SYNAPSE_TABLE_FOR_CROISSANT_LINKS} WHERE dataset_collection in ({','.join(dataset_collections_to_consider_for_deletion)})"
+            table_results = syn_client.tableQuery(
+                query=query, resultsAs="csv")
+            table_dataframe = table_results.asDataFrame(
+                rowIdAndVersionInIndex=False)
+            os.remove(table_results.filepath)
+            rows_to_delete = extract_synapse_rows_to_delete(
+                synapse_rows=table_dataframe,
+                combined_dataset_collection_and_datasets=combined_dataset_collection_and_datasets,
+            )
+
+            if rows_to_delete:
+                delete_out_of_date_from_synapse = context["params"]["delete_out_of_date_from_synapse"]
+                if delete_out_of_date_from_synapse:
+                    otel_logger.info(
+                        f"Deleting the following rows from Synapse: {rows_to_delete}")
+                    results_from_synapse = syn_client.tableQuery(
+                        query=f"SELECT ROW_ID, ROW_VERSION FROM {SYNAPSE_TABLE_FOR_CROISSANT_LINKS} WHERE ROW_ID in ({','.join(rows_to_delete)})", resultsAs="csv")
+                    syn_client.delete(results_from_synapse)
+                    os.remove(results_from_synapse.filepath)
+                else:
+                    otel_logger.info(
+                        f"Found rows to delete from Synapse, but not deleting due to `delete_out_of_date_from_synapse` param: {rows_to_delete}")
+            else:
+                otel_logger.info(
+                    "No rows to delete from Synapse. All rows are current.")
+            otel_tracer.span_processor.force_flush()
+            otel_logger.handlers[0].flush()
+            return None
+
+    @task
+    def query_and_push_croissant_file(root_carrier_context: Dict, dataset_id: str, dataset_version: int, dataset_collection: str, **context) -> None:
         """
         Query the Snowflake database for the dataset in order to perform the Croissant
         transformation and push the results to S3.
 
         Arguments:
             dataset_id: The ID of the dataset to query in Snowflake.
         """
-        with otel_tracer.start_as_current_span("query_snowflake_and_push_croissant_file", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+        with otel_tracer.start_as_current_span("query_and_push_croissant_file", context=TraceContextTextMapPropagator().extract(root_carrier_context)) as span:
+            push_to_s3 = context["params"]["push_results_to_s3"]
+            push_to_synapse = context["params"]["push_links_to_synapse"]
+            span.set_attribute("airflow.push_results_to_s3", push_to_s3)
+            span.set_attribute(
+                "airflow.push_links_to_synapse", push_to_synapse)
             if not dataset_id:
                 otel_logger.info("No dataset found")
                 otel_tracer.span_processor.force_flush()
                 otel_logger.handlers[0].flush()
                 return
 
-            span.set_attribute("airflow.dataset_id", f"syn{dataset_id}")
-
-            query = f"""
-            WITH entity_metadata AS (
-                SELECT
-                    items,
-                    name,
-                    ANNOTATIONS,
-                    CREATED_ON,
-                    VERSION_NUMBER,
-                    CONCAT('syn', id) as DATASET_SYNAPSE_ID
-                FROM synapse_data_warehouse.synapse.node_latest
-                WHERE id = {dataset_id}
-            ),
-            ids_of_files_belonging_to_dataset AS (
-                SELECT REPLACE(ITEM.value:entityId::STRING, 'syn', '') AS entity_id, ITEM.value:versionNumber::STRING as version_number
-                FROM entity_metadata,
-                LATERAL FLATTEN(input => entity_metadata.ITEMS) ITEM
-            ),
-            files_belonging_to_dataset AS (
-                SELECT
-                    nl.id,
-                    nl.version_number,
-                    vh.value AS VERSION_HISTORY_ENTRY,
-                    nl.ANNOTATIONS,
-                    nl.is_public,
-                    ids_of_files_belonging_to_dataset.entity_id,
-                    ids_of_files_belonging_to_dataset.version_number,
-                    entity_metadata.DATASET_SYNAPSE_ID
-                FROM synapse_data_warehouse.synapse.node_latest AS nl
-                JOIN ids_of_files_belonging_to_dataset
-                    ON nl.id = ids_of_files_belonging_to_dataset.entity_id
-                JOIN entity_metadata
-                    ON TRUE -- Ensures the `DATASET_SYNAPSE_ID` is included
-                CROSS JOIN LATERAL FLATTEN(input => nl.VERSION_HISTORY) vh
-                WHERE nl.is_public = TRUE
-                AND vh.value:versionNumber::STRING = ids_of_files_belonging_to_dataset.version_number
-            ),
-            distribution_pointing_to_files AS (
-                SELECT
-                    CONCAT('syn', id) as SYNAPSE_ID,
-                    DATASET_SYNAPSE_ID,
-                    OBJECT_CONSTRUCT(
-                        '@type', 'FileObject',
-                        '@id', CONCAT('syn', id),
-                        'name', CONCAT('syn', id),
-                        'description', CONCAT(
-                            'Data file associated with ', CONCAT('syn', id)),
-                        'contentUrl', CONCAT(
-                            'https://www.synapse.org/Synapse:', CONCAT('syn', id)),
-                        'encodingFormat', 'application/json',
-                        'md5', COALESCE(VERSION_HISTORY_ENTRY:contentMd5, 'unknown_md5'),
-                        'sha256', 'unknown'
-                    ) AS file_object
-                FROM files_belonging_to_dataset
-            ),
-            -- TODO: Annotation selection must be version dependent, however, the different versions of the annotations are not available in a non-deduplicated table
-            distinct_annotation_keys AS (
-                SELECT DATASET_SYNAPSE_ID, ARRAY_AGG(annotation_key) as annotation_keys FROM (
-                    SELECT
-                        DATASET_SYNAPSE_ID,
-                        key as annotation_key
-                    FROM files_belonging_to_dataset,
-                    LATERAL FLATTEN(input => ANNOTATIONS:annotations) AS annotation
-                    GROUP BY DATASET_SYNAPSE_ID, key
-                ) GROUP BY DATASET_SYNAPSE_ID
-            ),
-            fields_pointing_to_metadata AS (
-                SELECT
-                    DATASET_SYNAPSE_ID,
-                    OBJECT_CONSTRUCT(
-                        '@type', 'RecordSet',
-                        '@id', 'default',
-                        'name', 'default',
-                        'description', 'Metadata for the dataset',
-                        'field', ARRAY_AGG(
-                            OBJECT_CONSTRUCT(
-                                '@type', 'cr:Field',
-                                '@id', CONCAT('metadata', '/', value),
-                                'name', value,
-                                -- 'description', '',
-                                'dataType', 'sc:Text',
-                                'source', OBJECT_CONSTRUCT(
-                                    'fileObject', OBJECT_CONSTRUCT(
-                                        '@id', 'metadata'),
-                                    'extract', OBJECT_CONSTRUCT('column', value)
-                                )
-                            )
-                        )
-                    ) AS record_set
-                FROM distinct_annotation_keys,
-                LATERAL FLATTEN(input => distinct_annotation_keys.annotation_keys)
-                GROUP BY DATASET_SYNAPSE_ID
-            ),
-            croissant_metadata AS (
-                SELECT
-                    entity_metadata.DATASET_SYNAPSE_ID,
-                    entity_metadata.name,
-                    OBJECT_CONSTRUCT(
-                        '@context', OBJECT_CONSTRUCT(
-                            '@language', 'en',
-                            '@vocab', 'https://schema.org/',
-                            'citeAs', 'cr:citeAs',
-                            'column', 'cr:column',
-                            'conformsTo', 'dct:conformsTo',
-                            'cr', 'http://mlcommons.org/croissant/',
-                            'rai', 'http://mlcommons.org/croissant/RAI/',
-                            'data', OBJECT_CONSTRUCT(
-                                '@id', 'cr:data', '@type', '@json'),
-                            'dataType', OBJECT_CONSTRUCT(
-                                '@id', 'cr:dataType', '@type', '@vocab'),
-                            'dct', 'http://purl.org/dc/terms/',
-                            'examples', OBJECT_CONSTRUCT(
-                                '@id', 'cr:examples', '@type', '@json'),
-                            'extract', 'cr:extract',
-                            'field', 'cr:field',
-                            'fileProperty', 'cr:fileProperty',
-                            'fileObject', 'cr:fileObject',
-                            'fileSet', 'cr:fileSet',
-                            'format', 'cr:format',
-                            'includes', 'cr:includes',
-                            'isLiveDataset', 'cr:isLiveDataset',
-                            'jsonPath', 'cr:jsonPath',
-                            'key', 'cr:key',
-                            'md5', 'cr:md5',
-                            'parentField', 'cr:parentField',
-                            'path', 'cr:path',
-                            'recordSet', 'cr:recordSet',
-                            'references', 'cr:references',
-                            'regex', 'cr:regex',
-                            'repeated', 'cr:repeated',
-                            'replace', 'cr:replace',
-                            'sc', 'https://schema.org/',
-                            'separator', 'cr:separator',
-                            'source', 'cr:source',
-                            'subField', 'cr:subField',
-                            'transform', 'cr:transform'
-                        ),
-                        '@type', 'Dataset',
-                        'name', COALESCE(entity_metadata.name, 'Synapse Dataset'),
-                        'description', COALESCE(ANNOTATIONS:annotations:description:value[0], 'A dataset containing annotations for genomic data from Synapse.'),
-                        'url', CONCAT('https://www.synapse.org/Synapse:',
-                                    entity_metadata.DATASET_SYNAPSE_ID),
-                        'citation', COALESCE(ANNOTATIONS:annotations:citation:value[0], 'unknown_citation'),
-                        'datePublished', COALESCE(CREATED_ON, 'unknown_date'),
-                        'license', COALESCE(ANNOTATIONS:annotations:license:value[0], 'unknown_license'),
-                        'version', COALESCE(VERSION_NUMBER, 'unknown_version'),
-                        'distribution', ARRAY_PREPEND(ARRAY_AGG(distribution_pointing_to_files.file_object),
-                                            OBJECT_CONSTRUCT(
-                                                '@type', 'FileObject',
-                                                '@id', 'metadata',
-                                                'name', 'metadata',
-                                                'description', CONCAT(
-                                                    'Metadata associated with ', entity_metadata.DATASET_SYNAPSE_ID),
-                                                'contentUrl', CONCAT(
-                                                    'https://www.synapse.org/Synapse:', entity_metadata.DATASET_SYNAPSE_ID),
-                                                'encodingFormat', 'application/csv',
-                                                'md5', '',
-                                                'sha256', 'unknown'
-                                            )
-                                        ),
-                        'recordSet', fields_pointing_to_metadata.record_set
-                    ) AS metadata
-                FROM
-                    entity_metadata, fields_pointing_to_metadata
-                JOIN
-                    distribution_pointing_to_files
-                ON
-                    distribution_pointing_to_files.DATASET_SYNAPSE_ID = fields_pointing_to_metadata.DATASET_SYNAPSE_ID
-                GROUP BY
-                    entity_metadata.DATASET_SYNAPSE_ID,
-                    entity_metadata.name,
-                    fields_pointing_to_metadata.DATASET_SYNAPSE_ID,
-                    fields_pointing_to_metadata.record_set,
-                    entity_metadata.name,
-                    entity_metadata.ANNOTATIONS,
-                    entity_metadata.CREATED_ON,
-                    entity_metadata.VERSION_NUMBER
-            )
-
-            SELECT
-                metadata::STRING AS croissant_metadata,
-                DATASET_SYNAPSE_ID,
-                name
-            FROM
-                croissant_metadata;
-            """
-            snow_hook = SnowflakeHook(context["params"]["snowflake_developer_service_conn"])
-            ctx = snow_hook.get_conn()
-            cs = ctx.cursor()
-
-            otel_logger.info(
-                f"Performing query for dataset {dataset_id}")
+            span.set_attribute("airflow.dataset_id", f"{dataset_id}")
+            path_to_remove = None
+            syn_client = create_syn_client()
+            file_ids_and_versions_attached_to_dataset = []
 
             try:
path: dags/synapse-dataset-to-croissant.py
user: login: linglp
body: Great. Thanks for adding them. 
created_at: 2025-04-07T23:33:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,559 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
+        elif not file_md5s.empty and int(file.id.replace("syn", "")) in file_md5s["ID"].values:
+            file_md5 = file_md5s.loc[file_md5s["ID"] == int(
+                file.id.replace("syn", "")), "CONTENTMD5"].values[0]
+        else:
+            file_md5 = "unknown_md5"
+
+        distribution_files.append(
+            {
+                "@type": "FileObject",
+                "@id": f"{file.id}.{file.version_number}",
+                "name": f"{file.name}",
+                "description": f"Data file associated with {file.name}",
+                "contentUrl": f"https://www.synapse.org/Synapse:{file.id}.{file.version_number}",
+                "encodingFormat": "application/json",
+                "md5": file_md5,
+                "sha256": "unknown",
+            }
+        )
+    return distribution_files
+
+
+def construct_record_set_section_for_files(files_attached_to_dataset: List[File]) -> Dict[str, str]:
+    """
+    Construct the record set section for the files attached to the dataset. This is used
+    to extract the keys from the annotations of the files and create a Field for each
+    key.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+
+    Returns:
+        The record set section for the files attached to the dataset.
+    """
+    unique_annotation_keys = set()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        file_annotations = file.annotations.keys()
+        for key in file_annotations:
+            unique_annotation_keys.add(key)
+
+    metadata_fields = []
+    # Sort the set of unique_annotation_keys ignoring case, but keep the case in the result
+    unique_annotation_keys = sorted(unique_annotation_keys, key=str.casefold)
+
+    for key in unique_annotation_keys:
+        metadata_fields.append(
+            {
+                "@type": "Field",
+                "@id": f"metadata/{key}",
+                "name": key,
+                "description": "",
+                "dataType": "sc:Text",
+                "source": {
+                    "fileObject": {"@id": "metadata"},
+                    "extract": {"column": key}
+                }
+            }
+        )
+
+    return {
+        "@type": "RecordSet",
+        "@id": "default",
+        "name": "default",
+        "description": "Metadata for the dataset",
+        "field": metadata_fields
+    }
+
+
+def extract_s3_objects_to_delete(bucket_objects: List[str], dataset_collections_to_consider_for_deletion: List[str], combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the S3 objects to delete from the S3 bucket. This is done by retrieving
+    all of the objects in the S3 bucket and checking if the object matches one of the
+    datasets that will be kept after the full DAG as been run. Every object in the S3
+    bucket that matches the pattern `_syn####.###_` and
+    `_datasetCollection_syn####_` will be considered for deletion. The DAG param
+    `dataset_collections_for_cleanup` is used to filter the objects that will be
+    considered for deletion as the second filter.
+
+    The last filter enforces that each object in the S3 bucket must match a dataset
+    id, dataset version, dataset collection, and dataset name in the
+    `combined_dataset_collection_and_datasets` list. If the object does not match
+    any of the objects in the list, then it will be marked for deletion.
+
+    Arguments:
path: dags/synapse-dataset-to-croissant.py
user: login: linglp
body: here this is also missing a docstring for parameter `combined_dataset_collection_and_datasets`? 
created_at: 2025-04-07T23:42:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,559 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
+        elif not file_md5s.empty and int(file.id.replace("syn", "")) in file_md5s["ID"].values:
+            file_md5 = file_md5s.loc[file_md5s["ID"] == int(
+                file.id.replace("syn", "")), "CONTENTMD5"].values[0]
+        else:
+            file_md5 = "unknown_md5"
+
+        distribution_files.append(
+            {
+                "@type": "FileObject",
+                "@id": f"{file.id}.{file.version_number}",
+                "name": f"{file.name}",
+                "description": f"Data file associated with {file.name}",
+                "contentUrl": f"https://www.synapse.org/Synapse:{file.id}.{file.version_number}",
+                "encodingFormat": "application/json",
+                "md5": file_md5,
+                "sha256": "unknown",
+            }
+        )
+    return distribution_files
+
+
+def construct_record_set_section_for_files(files_attached_to_dataset: List[File]) -> Dict[str, str]:
+    """
+    Construct the record set section for the files attached to the dataset. This is used
+    to extract the keys from the annotations of the files and create a Field for each
+    key.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+
+    Returns:
+        The record set section for the files attached to the dataset.
+    """
+    unique_annotation_keys = set()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        file_annotations = file.annotations.keys()
+        for key in file_annotations:
+            unique_annotation_keys.add(key)
+
+    metadata_fields = []
+    # Sort the set of unique_annotation_keys ignoring case, but keep the case in the result
+    unique_annotation_keys = sorted(unique_annotation_keys, key=str.casefold)
+
+    for key in unique_annotation_keys:
+        metadata_fields.append(
+            {
+                "@type": "Field",
+                "@id": f"metadata/{key}",
+                "name": key,
+                "description": "",
+                "dataType": "sc:Text",
+                "source": {
+                    "fileObject": {"@id": "metadata"},
+                    "extract": {"column": key}
+                }
+            }
+        )
+
+    return {
+        "@type": "RecordSet",
+        "@id": "default",
+        "name": "default",
+        "description": "Metadata for the dataset",
+        "field": metadata_fields
+    }
+
+
+def extract_s3_objects_to_delete(bucket_objects: List[str], dataset_collections_to_consider_for_deletion: List[str], combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the S3 objects to delete from the S3 bucket. This is done by retrieving
+    all of the objects in the S3 bucket and checking if the object matches one of the
+    datasets that will be kept after the full DAG as been run. Every object in the S3
+    bucket that matches the pattern `_syn####.###_` and
+    `_datasetCollection_syn####_` will be considered for deletion. The DAG param
+    `dataset_collections_for_cleanup` is used to filter the objects that will be
+    considered for deletion as the second filter.
+
+    The last filter enforces that each object in the S3 bucket must match a dataset
+    id, dataset version, dataset collection, and dataset name in the
+    `combined_dataset_collection_and_datasets` list. If the object does not match
+    any of the objects in the list, then it will be marked for deletion.
+
+    Arguments:
+        bucket_objects: The list of objects in the S3 bucket.
+        dataset_collections_to_consider_for_deletion: The dataset collections to
+            consider for deletion. This is passed in from the
+            `dataset_collections_for_cleanup` DAG parameter.
+    """
+    objects_to_delete = []
path: dags/synapse-dataset-to-croissant.py
user: login: linglp
body: can we also add more comments to this function? 
created_at: 2025-04-07T23:43:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,559 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
+        elif not file_md5s.empty and int(file.id.replace("syn", "")) in file_md5s["ID"].values:
+            file_md5 = file_md5s.loc[file_md5s["ID"] == int(
+                file.id.replace("syn", "")), "CONTENTMD5"].values[0]
+        else:
+            file_md5 = "unknown_md5"
+
+        distribution_files.append(
+            {
+                "@type": "FileObject",
+                "@id": f"{file.id}.{file.version_number}",
+                "name": f"{file.name}",
+                "description": f"Data file associated with {file.name}",
+                "contentUrl": f"https://www.synapse.org/Synapse:{file.id}.{file.version_number}",
+                "encodingFormat": "application/json",
+                "md5": file_md5,
+                "sha256": "unknown",
+            }
+        )
+    return distribution_files
+
+
+def construct_record_set_section_for_files(files_attached_to_dataset: List[File]) -> Dict[str, str]:
+    """
+    Construct the record set section for the files attached to the dataset. This is used
+    to extract the keys from the annotations of the files and create a Field for each
+    key.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+
+    Returns:
+        The record set section for the files attached to the dataset.
+    """
+    unique_annotation_keys = set()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        file_annotations = file.annotations.keys()
+        for key in file_annotations:
+            unique_annotation_keys.add(key)
+
+    metadata_fields = []
+    # Sort the set of unique_annotation_keys ignoring case, but keep the case in the result
+    unique_annotation_keys = sorted(unique_annotation_keys, key=str.casefold)
+
+    for key in unique_annotation_keys:
+        metadata_fields.append(
+            {
+                "@type": "Field",
+                "@id": f"metadata/{key}",
+                "name": key,
+                "description": "",
+                "dataType": "sc:Text",
+                "source": {
+                    "fileObject": {"@id": "metadata"},
+                    "extract": {"column": key}
+                }
+            }
+        )
+
+    return {
+        "@type": "RecordSet",
+        "@id": "default",
+        "name": "default",
+        "description": "Metadata for the dataset",
+        "field": metadata_fields
+    }
+
+
+def extract_s3_objects_to_delete(bucket_objects: List[str], dataset_collections_to_consider_for_deletion: List[str], combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the S3 objects to delete from the S3 bucket. This is done by retrieving
+    all of the objects in the S3 bucket and checking if the object matches one of the
+    datasets that will be kept after the full DAG as been run. Every object in the S3
+    bucket that matches the pattern `_syn####.###_` and
+    `_datasetCollection_syn####_` will be considered for deletion. The DAG param
+    `dataset_collections_for_cleanup` is used to filter the objects that will be
+    considered for deletion as the second filter.
+
+    The last filter enforces that each object in the S3 bucket must match a dataset
+    id, dataset version, dataset collection, and dataset name in the
+    `combined_dataset_collection_and_datasets` list. If the object does not match
+    any of the objects in the list, then it will be marked for deletion.
+
+    Arguments:
+        bucket_objects: The list of objects in the S3 bucket.
+        dataset_collections_to_consider_for_deletion: The dataset collections to
+            consider for deletion. This is passed in from the
+            `dataset_collections_for_cleanup` DAG parameter.
+    """
+    objects_to_delete = []
path: dags/synapse-dataset-to-croissant.py
user: login: BryanFauble
body: Re-read over the docstring of this function and let me know what your thoughts are. The code is relatively straight forward
created_at: 2025-04-07T23:45:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,559 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
+        elif not file_md5s.empty and int(file.id.replace("syn", "")) in file_md5s["ID"].values:
+            file_md5 = file_md5s.loc[file_md5s["ID"] == int(
+                file.id.replace("syn", "")), "CONTENTMD5"].values[0]
+        else:
+            file_md5 = "unknown_md5"
+
+        distribution_files.append(
+            {
+                "@type": "FileObject",
+                "@id": f"{file.id}.{file.version_number}",
+                "name": f"{file.name}",
+                "description": f"Data file associated with {file.name}",
+                "contentUrl": f"https://www.synapse.org/Synapse:{file.id}.{file.version_number}",
+                "encodingFormat": "application/json",
+                "md5": file_md5,
+                "sha256": "unknown",
+            }
+        )
+    return distribution_files
+
+
+def construct_record_set_section_for_files(files_attached_to_dataset: List[File]) -> Dict[str, str]:
+    """
+    Construct the record set section for the files attached to the dataset. This is used
+    to extract the keys from the annotations of the files and create a Field for each
+    key.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+
+    Returns:
+        The record set section for the files attached to the dataset.
+    """
+    unique_annotation_keys = set()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        file_annotations = file.annotations.keys()
+        for key in file_annotations:
+            unique_annotation_keys.add(key)
+
+    metadata_fields = []
+    # Sort the set of unique_annotation_keys ignoring case, but keep the case in the result
+    unique_annotation_keys = sorted(unique_annotation_keys, key=str.casefold)
+
+    for key in unique_annotation_keys:
+        metadata_fields.append(
+            {
+                "@type": "Field",
+                "@id": f"metadata/{key}",
+                "name": key,
+                "description": "",
+                "dataType": "sc:Text",
+                "source": {
+                    "fileObject": {"@id": "metadata"},
+                    "extract": {"column": key}
+                }
+            }
+        )
+
+    return {
+        "@type": "RecordSet",
+        "@id": "default",
+        "name": "default",
+        "description": "Metadata for the dataset",
+        "field": metadata_fields
+    }
+
+
+def extract_s3_objects_to_delete(bucket_objects: List[str], dataset_collections_to_consider_for_deletion: List[str], combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the S3 objects to delete from the S3 bucket. This is done by retrieving
+    all of the objects in the S3 bucket and checking if the object matches one of the
+    datasets that will be kept after the full DAG as been run. Every object in the S3
+    bucket that matches the pattern `_syn####.###_` and
+    `_datasetCollection_syn####_` will be considered for deletion. The DAG param
+    `dataset_collections_for_cleanup` is used to filter the objects that will be
+    considered for deletion as the second filter.
+
+    The last filter enforces that each object in the S3 bucket must match a dataset
+    id, dataset version, dataset collection, and dataset name in the
+    `combined_dataset_collection_and_datasets` list. If the object does not match
+    any of the objects in the list, then it will be marked for deletion.
+
+    Arguments:
+        bucket_objects: The list of objects in the S3 bucket.
+        dataset_collections_to_consider_for_deletion: The dataset collections to
+            consider for deletion. This is passed in from the
+            `dataset_collections_for_cleanup` DAG parameter.
+    """
+    objects_to_delete = []
path: dags/synapse-dataset-to-croissant.py
user: login: linglp
body: Thanks for updating the docstring. I understand the most parts, but I am curious why it is important to enforce the last filter? why each object in the S3 bucket must match a dataset id, dataset version, dataset collection, and dataset name? is it for ensuring that we delete S3 objects that were no longer present in the dataset collections? I know that you have mentioned that in the PR description, but could you add a comment in the code or clarify in the docstring? I am thinking referencing the PR description might not be as convenient as looking at the code comment. 
created_at: 2025-04-08T00:59:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,559 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
+        elif not file_md5s.empty and int(file.id.replace("syn", "")) in file_md5s["ID"].values:
+            file_md5 = file_md5s.loc[file_md5s["ID"] == int(
+                file.id.replace("syn", "")), "CONTENTMD5"].values[0]
+        else:
+            file_md5 = "unknown_md5"
+
+        distribution_files.append(
+            {
+                "@type": "FileObject",
+                "@id": f"{file.id}.{file.version_number}",
+                "name": f"{file.name}",
+                "description": f"Data file associated with {file.name}",
+                "contentUrl": f"https://www.synapse.org/Synapse:{file.id}.{file.version_number}",
+                "encodingFormat": "application/json",
+                "md5": file_md5,
+                "sha256": "unknown",
+            }
+        )
+    return distribution_files
+
+
+def construct_record_set_section_for_files(files_attached_to_dataset: List[File]) -> Dict[str, str]:
+    """
+    Construct the record set section for the files attached to the dataset. This is used
+    to extract the keys from the annotations of the files and create a Field for each
+    key.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+
+    Returns:
+        The record set section for the files attached to the dataset.
+    """
+    unique_annotation_keys = set()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        file_annotations = file.annotations.keys()
+        for key in file_annotations:
+            unique_annotation_keys.add(key)
+
+    metadata_fields = []
+    # Sort the set of unique_annotation_keys ignoring case, but keep the case in the result
+    unique_annotation_keys = sorted(unique_annotation_keys, key=str.casefold)
+
+    for key in unique_annotation_keys:
+        metadata_fields.append(
+            {
+                "@type": "Field",
+                "@id": f"metadata/{key}",
+                "name": key,
+                "description": "",
+                "dataType": "sc:Text",
+                "source": {
+                    "fileObject": {"@id": "metadata"},
+                    "extract": {"column": key}
+                }
+            }
+        )
+
+    return {
+        "@type": "RecordSet",
+        "@id": "default",
+        "name": "default",
+        "description": "Metadata for the dataset",
+        "field": metadata_fields
+    }
+
+
+def extract_s3_objects_to_delete(bucket_objects: List[str], dataset_collections_to_consider_for_deletion: List[str], combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the S3 objects to delete from the S3 bucket. This is done by retrieving
+    all of the objects in the S3 bucket and checking if the object matches one of the
+    datasets that will be kept after the full DAG as been run. Every object in the S3
+    bucket that matches the pattern `_syn####.###_` and
+    `_datasetCollection_syn####_` will be considered for deletion. The DAG param
+    `dataset_collections_for_cleanup` is used to filter the objects that will be
+    considered for deletion as the second filter.
+
+    The last filter enforces that each object in the S3 bucket must match a dataset
+    id, dataset version, dataset collection, and dataset name in the
+    `combined_dataset_collection_and_datasets` list. If the object does not match
+    any of the objects in the list, then it will be marked for deletion.
+
+    Arguments:
+        bucket_objects: The list of objects in the S3 bucket.
+        dataset_collections_to_consider_for_deletion: The dataset collections to
+            consider for deletion. This is passed in from the
+            `dataset_collections_for_cleanup` DAG parameter.
+    """
+    objects_to_delete = []
path: dags/synapse-dataset-to-croissant.py
user: login: linglp
body: I think I understand now after reading more about the doc string in delete_non_current_croissant_file_in_s3 function. We could keep the code there as it is. Thank you 
created_at: 2025-04-08T01:09:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -165,17 +199,559 @@ def set_up_logging() -> Logger:
 
     handler = LoggingHandler(level=NOTSET,
                              logger_provider=logger_provider)
-    getLogger(__name__).addHandler(handler)
-    return getLogger(__name__)
+    logger = getLogger()
+    logger.addHandler(handler)
+    return logger
 
 
 otel_tracer = set_up_tracing()
 otel_logger = set_up_logging()
 
 
+# TODO: Remove this on the next release of py-orca
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks-Workflows/py-orca/pull/50
+def create_client(self) -> Synapse:
+    """
+    See original create_client method in for more details.
+    """
+    client = Synapse(silent=True, skip_checks=True)
+    auth_token = self.config.auth_token
+
+    if auth_token is None:
+        message = f"Config ({self.config}) is missing auth_token."
+        raise ConfigError(message)
+
+    # Ignore authentication error (leave that to `test_client_request`)
+    try:
+        client.login(authToken=auth_token)
+    except SynapseAuthenticationError:
+        pass
+
+    return client
+
+
+SynapseClientFactory.create_client = create_client
+
+# TODO: Remove this on the next > 4.7.0 release of the Synapse Python Client
+# This is a temporary hack to include the changes from: https://github.com/Sage-Bionetworks/synapsePythonClient/pull/1188
+# The hack is used here because the current SYNPY client does not have an HTTP timeout
+# for requests to Synapse. As a result and due to the significant number of HTTP calls that
+# occur during the DAG, the DAG can stall and never return due to the requests library.
+# https://requests.readthedocs.io/en/latest/user/advanced/#timeouts
+
+
+def _rest_call_replacement(
+    self,
+    method,
+    uri,
+    data,
+    endpoint,
+    headers,
+    retryPolicy,
+    requests_session,
+    **kwargs,
+):
+    """
+    See original _rest_call method in the Synapse client for more details.
+    """
+    self.logger.debug(f"Sending {method} request to {uri}")
+    uri, headers = self._build_uri_and_headers(
+        uri, endpoint=endpoint, headers=headers
+    )
+
+    retryPolicy = self._build_retry_policy(retryPolicy)
+    requests_session = requests_session or self._requests_session
+
+    auth = kwargs.pop("auth", self.credentials)
+    requests_method_fn = getattr(requests_session, method)
+    response = with_retry(
+        lambda: requests_method_fn(
+            uri,
+            data=data,
+            headers=headers,
+            auth=auth,
+            timeout=70,
+            **kwargs,
+        ),
+        verbose=self.debug,
+        **retryPolicy,
+    )
+    self._handle_synapse_http_error(response)
+    return response
+
+
+def create_syn_client() -> Synapse:
+    """
+    Create a Synapse client that can be used to query Synapse.
+
+    Returns:
+        The Synapse client.
+    """
+    syn_client: Synapse = Synapse(skip_checks=True)
+    syn_client._rest_call = MethodType(_rest_call_replacement, syn_client)
+    assert syn_client.credentials is None, "Synapse client is not logged out"
+    return syn_client
+
+
+def get_file_instances(
+    synapse_files: List[Dict[str, str]], syn_client: Synapse
+) -> List[File]:
+    """
+    Get the file instances for the given list of files.
+
+    Arguments:
+        synapse_files: The list of files to get the file instances for.
+        syn_client: The Synapse client to use to get the file instances.
+
+    Returns:
+        The list of file instances.
+    """
+    file_metadata: List[File] = []
+    for synapse_file in synapse_files:
+        try:
+            file_instance = File(id=synapse_file.get(
+                "file_id"), version_number=synapse_file.get("file_version"), download_file=False).get(synapse_client=syn_client)
+            file_metadata.append(file_instance)
+        except SynapseHTTPError as ex:
+            if "404 Client Error" in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} not found in Synapse.")
+            else:
+                raise ex
+        except SynapseAuthenticationError as ex:
+            if "You are not logged in and do not have access to a requested resource." in str(ex):
+                otel_logger.warning(
+                    f"File {synapse_file.get('file_id')} is not anonymously accessible.")
+            else:
+                raise ex
+    return file_metadata
+
+
+def construct_distribution_section_for_files(files_attached_to_dataset: List[File], **context) -> List[Dict[str, str]]:
+    """
+    Construct the distribution section for the files attached to the dataset. This is
+    used to extract various metadata from the files and create a FileObject for each
+    file.
+
+
+    When we do not have the content_md5 for a file, we need to query Snowflake to get
+    the content_md5 for the file. This is done by querying the synapse_data_warehouse
+    database in Snowflake.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+        context: The context of the DAG run.
+
+    Returns:
+        The distribution section for the files attached to the dataset.
+    """
+    distribution_files = []
+    files_to_find_md5_in_snowflake = {}
+
+    ids_of_files = []
+    id_and_version_pairs = []
+    for file in files_attached_to_dataset:
+        file: File = file
+        if not file.file_handle or not file.file_handle.content_md5:
+            modified_file_id = int(file.id.replace("syn", ""))
+            ids_of_files.append(modified_file_id)
+            id_and_version_pairs.append(modified_file_id)
+            id_and_version_pairs.append(file.version_number)
+            files_to_find_md5_in_snowflake[int(file.id.replace(
+                "syn", ""))] = file.version_number
+
+    file_md5s = None
+    if ids_of_files:
+        snow_hook = SnowflakeHook(
+            context["params"]["snowflake_developer_service_conn"])
+        ctx = snow_hook.get_conn()
+        cs = ctx.cursor()
+
+        query = f"""
+        WITH version_data AS (
+            SELECT
+                nl.id,
+                flattened.value:versionNumber::int AS versionNumber,
+                flattened.value:contentMd5::string AS contentMd5
+            FROM synapse_data_warehouse.synapse.node_latest AS nl,
+            LATERAL FLATTEN(input => nl.VERSION_HISTORY) AS flattened
+            WHERE nl.id IN ({', '.join(['%s'] * len(ids_of_files))})
+            AND nl.is_public = TRUE
+        )
+        SELECT
+            vd.id,
+            vd.versionNumber,
+            vd.contentMd5
+        FROM version_data AS vd
+        WHERE (vd.id, vd.versionNumber) IN ({', '.join(['(%s, %s)'] * (len(id_and_version_pairs)//2))});
+        """
+        try:
+            cs.execute(
+                query,
+                (ids_of_files + id_and_version_pairs),
+            )
+            file_md5s = cs.fetch_pandas_all()
+        finally:
+            cs.close()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        if file.file_handle and file.file_handle.content_md5:
+            file_md5 = file.file_handle.content_md5
+        elif not file_md5s.empty and int(file.id.replace("syn", "")) in file_md5s["ID"].values:
+            file_md5 = file_md5s.loc[file_md5s["ID"] == int(
+                file.id.replace("syn", "")), "CONTENTMD5"].values[0]
+        else:
+            file_md5 = "unknown_md5"
+
+        distribution_files.append(
+            {
+                "@type": "FileObject",
+                "@id": f"{file.id}.{file.version_number}",
+                "name": f"{file.name}",
+                "description": f"Data file associated with {file.name}",
+                "contentUrl": f"https://www.synapse.org/Synapse:{file.id}.{file.version_number}",
+                "encodingFormat": "application/json",
+                "md5": file_md5,
+                "sha256": "unknown",
+            }
+        )
+    return distribution_files
+
+
+def construct_record_set_section_for_files(files_attached_to_dataset: List[File]) -> Dict[str, str]:
+    """
+    Construct the record set section for the files attached to the dataset. This is used
+    to extract the keys from the annotations of the files and create a Field for each
+    key.
+
+    Arguments:
+        files_attached_to_dataset: The list of files attached to the dataset.
+
+    Returns:
+        The record set section for the files attached to the dataset.
+    """
+    unique_annotation_keys = set()
+
+    for file in files_attached_to_dataset:
+        file: File = file
+        file_annotations = file.annotations.keys()
+        for key in file_annotations:
+            unique_annotation_keys.add(key)
+
+    metadata_fields = []
+    # Sort the set of unique_annotation_keys ignoring case, but keep the case in the result
+    unique_annotation_keys = sorted(unique_annotation_keys, key=str.casefold)
+
+    for key in unique_annotation_keys:
+        metadata_fields.append(
+            {
+                "@type": "Field",
+                "@id": f"metadata/{key}",
+                "name": key,
+                "description": "",
+                "dataType": "sc:Text",
+                "source": {
+                    "fileObject": {"@id": "metadata"},
+                    "extract": {"column": key}
+                }
+            }
+        )
+
+    return {
+        "@type": "RecordSet",
+        "@id": "default",
+        "name": "default",
+        "description": "Metadata for the dataset",
+        "field": metadata_fields
+    }
+
+
+def extract_s3_objects_to_delete(bucket_objects: List[str], dataset_collections_to_consider_for_deletion: List[str], combined_dataset_collection_and_datasets: List[Dict[str, str]]) -> List[str]:
+    """
+    Extract the S3 objects to delete from the S3 bucket. This is done by retrieving
+    all of the objects in the S3 bucket and checking if the object matches one of the
+    datasets that will be kept after the full DAG as been run. Every object in the S3
+    bucket that matches the pattern `_syn####.###_` and
+    `_datasetCollection_syn####_` will be considered for deletion. The DAG param
+    `dataset_collections_for_cleanup` is used to filter the objects that will be
+    considered for deletion as the second filter.
+
+    The last filter enforces that each object in the S3 bucket must match a dataset
+    id, dataset version, dataset collection, and dataset name in the
+    `combined_dataset_collection_and_datasets` list. If the object does not match
+    any of the objects in the list, then it will be marked for deletion.
+
+    Arguments:
+        bucket_objects: The list of objects in the S3 bucket.
+        dataset_collections_to_consider_for_deletion: The dataset collections to
+            consider for deletion. This is passed in from the
+            `dataset_collections_for_cleanup` DAG parameter.
+    """
+    objects_to_delete = []
path: dags/synapse-dataset-to-croissant.py
user: login: thomasyu888
body: @linglp thanks for the thoughtful review. My 2 cents is that module docstring covers this information on line 12 which is the requirement to remove the json from the S3 bucket for two scenarios:

1. Datasets removed from the dataset collection
2. Datasets made private from public (since these croissant files are publicly accessible)

I'll defer to Bryan if he would like to add more information but imo, we also don't need to over document.


created_at: 2025-04-08T01:12:18Z
author_association: CONTRIBUTOR

PR comments:
user: login: sonarqubecloud[bot]
created_at: 2025-04-09T20:59:48Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=Sage-Bionetworks-Workflows_orca-recipes&pullRequest=83) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [55 New issues](https://sonarcloud.io/project/issues?id=Sage-Bionetworks-Workflows_orca-recipes&pullRequest=83&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [5 Accepted issues](https://sonarcloud.io/project/issues?id=Sage-Bionetworks-Workflows_orca-recipes&pullRequest=83&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [4 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=Sage-Bionetworks-Workflows_orca-recipes&pullRequest=83&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=Sage-Bionetworks-Workflows_orca-recipes&pullRequest=83&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=Sage-Bionetworks-Workflows_orca-recipes&pullRequest=83&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=Sage-Bionetworks-Workflows_orca-recipes&pullRequest=83)


author_association: NONE