Repository: bridge2ai/b2ai-standards-registry
Pull request #303: Issue 254 nonempty table versions
Author: login: Sigfried
Created: 2025-05-05T21:03:34Z | Merged: 2025-05-16T19:21:31Z

WIP


Diff:
diff --git a/.github/scripts/update_synapse_tables.py b/.github/scripts/update_synapse_tables.py
index 884ce399..e37423cd 100644
--- a/.github/scripts/update_synapse_tables.py
+++ b/.github/scripts/update_synapse_tables.py
@@ -1,79 +1,5 @@
-import csv
-import json
 import sys
-from synapseclient import Synapse, Table
-import os
-import pandas as pd
-
-
-# the paths detected as changes from the "Get changed files" job mapped to their corresponding synapse table ids
-PATHS_TO_IDS = {
-    "project/data/DataStandardOrTool.json": "syn63096833",
-    "project/data/DataSubstrate.json": "syn63096834",
-    "project/data/DataTopic.json": "syn63096835",
-    "project/data/Organization.json": "syn63096836",
-    "project/data/UseCase.json": "syn63096837",
-    "project/data/DataSet.json": "syn66330217",
-}
-
-
-def delete_table_rows(syn: Synapse, table_id: str) -> None:
-    """Delete all the rows for a table, given its id
-    :param syn: synapse client
-    :param table_id: table id to delete rows for
-    """
-    print(f"Deleting rows from table {table_id}")
-    results = syn.tableQuery(f"select * from {table_id}")
-    syn.delete(results)
-    print("Finished deleting rows")
-
-
-def populate_table(syn: Synapse, update_file: str, table_id: str) -> None:
-    """Populate the table with updated data
-    :param syn: synapse client
-    :param update_file: path for json file containing data to populate the table
-    :param table_id: synapse id for table to populate
-    """
-    with open(update_file, "r") as file:
-        data = json.load(file)
-    # each json file begins with a key that maps to the list of records, so we're accessing that list here
-    data = next(iter(data.values()), [])
-    if isinstance(data, list):
-        df = pd.DataFrame(data=data)
-        print(f"Populating table for file: {update_file}")
-        table = syn.store(Table(table_id, df))
-        print("Finished populating table")
-    else:
-        print("Could not get list of data from json file")
-
-
-def main():
-    try:
-        changed_files = sys.argv[1:]
-
-        if not changed_files:
-            print("No relevant files passed to the script.")
-            return
-
-        print("Creating synapse client...")
-        syn = Synapse()
-        auth_token = os.getenv("SYNAPSE_AUTH_TOKEN")
-        if not auth_token:
-            raise ValueError("SYNAPSE_AUTH_TOKEN environment variable is not set")
-        print("Signing in...")
-        syn.login(authToken=auth_token)
-
-        for changed_file in changed_files:
-            table_id = PATHS_TO_IDS.get(changed_file)
-            print(f"Creating snapshot for table {changed_file}")
-            syn.create_snapshot_version(table_id)
-            delete_table_rows(syn, table_id)
-            populate_table(syn, changed_file, table_id)
-
-    except Exception as e:
-        print(f"An error occurred when trying to update synapse tables: {e}")
-        sys.exit(1)
-
+from scripts.analyze_and_update_synapse_tables import analyze_and_update
 
 if __name__ == "__main__":
-    main()
+    analyze_and_update(sys.argv[1:])
diff --git a/.github/workflows/main.yaml b/.github/workflows/main.yaml
index 08999676..1c207d5a 100644
--- a/.github/workflows/main.yaml
+++ b/.github/workflows/main.yaml
@@ -44,8 +44,9 @@ jobs:
       #----------------------------------------------
       #    Run tests
       #----------------------------------------------
-      - name: Run validator tests
-        run: make test
+      #   commenting out because there currently are no tests
+      # - name: Run validator tests
+      #   run: make test
 
       #----------------------------------------------
       #    Validate data
diff --git a/scripts/README.md b/scripts/README.md
index 76a5b5e0..1495a439 100644
--- a/scripts/README.md
+++ b/scripts/README.md
@@ -43,20 +43,22 @@ The scripts in this folder are designed to:
 
 Each script is intended to be run individually. Here’s how to use them:
 
-### Example Script: modify_synapse_schema.py
+### Example Script: analyze_and_update_synapse_tables.py
 
-**Description:** Sets up and manages table schemas in Synapse, defining the column structure for each table.
+**Description:** Uploads tables from registry .json files to Synapse. Synapse table
+schemas will be based on the data uploaded.
 
 **Run:**
 
-you need to run all fucntions with a poetry run command at the start as such:
+You should run all functions with a poetry run command at the start as such:
 
 ```bash
 poetry run {command here}
 ```
 
+For instance, to upload the Organization and DataTopic tables, run
 ```bash
-poetry run python scripts/modify_synapse_schema.py
+poetry run python -m scripts.analyze_and_update_synapse_tables -t Organization DataTopic
 ```
 
 ### Script: format_yaml.py
diff --git a/scripts/analyze_and_update_synapse_tables.py b/scripts/analyze_and_update_synapse_tables.py
new file mode 100644
index 00000000..70a7ce28
--- /dev/null
+++ b/scripts/analyze_and_update_synapse_tables.py
@@ -0,0 +1,154 @@
+import json
+import sys
+from argparse import ArgumentParser
+from synapseclient import Synapse, Table, Schema, Column
+from pandas.api.types import infer_dtype
+import pandas as pd
+from scripts.utils import initialize_synapse, create_or_clear_table, PROJECT_ID
+
+
+# the paths detected as changes from the "Get changed files" job mapped to their corresponding synapse table ids
+PATHS_TO_IDS = {
+    "project/data/DataStandardOrTool.json": "syn63096833",
+    "project/data/DataSubstrate.json": "syn63096834",
+    "project/data/DataTopic.json": "syn63096835",
+    "project/data/Organization.json": "syn63096836",
+    "project/data/UseCase.json": "syn63096837",
+    "project/data/DataSet.json": "syn66330217",
+}
+
+def file_path_to_table_name(path: str) -> str:
+    return path.split('/')[-1].split('.')[0]
+
+TABLE_IDS = {file_path_to_table_name(p): PATHS_TO_IDS[p] for p in PATHS_TO_IDS.keys()}
+
+def populate_table(syn: Synapse, update_file: str, table_id: str) -> None:
+    """Populate the table with updated data
+    :param syn: synapse client
+    :param update_file: path for json file containing data to populate the table
+    :param table_id: synapse id for table to populate
+    """
+    with open(update_file, "r") as file:
+        data = json.load(file)
+    # each json file begins with a key that maps to the list of records, so we're accessing that list here
+    data = next(iter(data.values()), [])
+
+    if not isinstance(data, list):
+        print("Could not get list of data from json file")
+        return
+
+    df = pd.DataFrame(data=data)
+
+    cols = get_col_defs(df)
+    coldefs = [Column(**col) for col in cols.values()]
+
+    table_name = file_path_to_table_name(update_file)
+
+    # moved this from main() so we don't delete rows till the last minute
+    print(f"Creating snapshot and clearing table {update_file}")
+    create_or_clear_table(syn, table_name)
+
+    schema = Schema(name=table_name, columns=coldefs, parent=PROJECT_ID)
+    table = Table(name=table_name, parent_id=PROJECT_ID, schema=schema, values=df)
+    table.tableId = table_id
+
+    print(f"Populating table for file: {update_file}")
+    table = syn.store(table)
+    print("Finished populating table")
+
+
+# synapse defaults
+default_maximumSize = 50    # ignoring this because some tables will be too big if defaulting to it
+                            #   instead, make maximumSize just big enough to fit the longest column value
+default_maximumListLength = 100
+
+def get_col_defs(new_data_df):
+    is_list_cols = (new_data_df.map(type).astype(str) == "<class 'list'>").any()
+    list_cols = set(is_list_cols[is_list_cols == True].index)
+    scalar_types = {c: infer_dtype(new_data_df[c], skipna=True).upper() for c in new_data_df.columns} # infer_dtype gives 'mixed' for list types
+    # assuming all list columns are string lists, at least for now
+    get_col_type = lambda col_name: 'STRING_LIST' if col_name in list_cols else scalar_types[col_name]
+
+    new_cols = {col_name: {'name': col_name, 'columnType': get_col_type(col_name)} for col_name in new_data_df.columns}
+
+
+    for col_name in new_cols:
+        new_col = new_cols[col_name]
+        actual_max_size = 0
+
+        if new_col['columnType'].endswith('_LIST'):
+            actual_max_list_len = 0
+            for value in new_data_df[col_name].dropna():
+                actual_max_list_len = max(actual_max_list_len, len(value))
+                # Find longest string in this list
+                if value:  # Check if list is not empty
+                    item_lengths = [len(str(item)) for item in value]
+                    max_item_in_this_list = max(item_lengths) if item_lengths else 0
+                    actual_max_size = max(actual_max_size, max_item_in_this_list)
+            if actual_max_list_len > default_maximumListLength:
+                new_col['maximumListLength'] = int(actual_max_list_len)
+        else:
+            actual_max_size = new_data_df[col_name].astype(str).str.len().max()
+            if new_col['columnType'] == 'STRING':
+                if actual_max_size > 2000:
+                    new_col['columnType'] = 'LARGETEXT'
+                elif actual_max_size > 1000:
+                    new_col['columnType'] = 'MEDIUMTEXT'
+        new_col['maximumSize'] = int(actual_max_size)
+    return new_cols
+
+
+def analyze_and_update(files, all=False, table_names=False):
+    try:
+        if all:
+            files = list(PATHS_TO_IDS.keys())
+
+        if table_names:
+            files = [f"project/data/{table_name}.json" for table_name in files]
+
+        for file in files:
+            if file not in PATHS_TO_IDS:
+                raise ValueError(f"File is not a valid table: {file}")
+
+        if not files:
+            raise ValueError(f"No files given")
+
+        print("Signing in...")
+        syn = initialize_synapse()
+
+        for file in files:
+            table_id = PATHS_TO_IDS.get(file)
+            populate_table(syn, file, table_id)
+
+    except Exception as e:
+        print(f"An error occurred when trying to update synapse tables: {e}")
+        sys.exit(1)
+
+
+def cli():
+    """Command line interface"""
+    parser = ArgumentParser(
+        description='Uploads registry tables to Synapse, analyzing data to create appropriate schema. '
+                    f'Possible tables are {', '.join(TABLE_IDS.keys())}.'
+    )
+
+    parser.add_argument('-a', '--all', action='store_true', help='Upload all tables')
+
+    parser.add_argument('-t', '--table-names', action='store_true', default=False,
+                        help='Just provide table names. They will be converted to /project/data/<table-name>.json')
+
+    parser.add_argument('files', nargs='*', help='Files to upload. Should include path from root. Usually /project/data/')
+
+    args = parser.parse_args()
+
+    if not args.all and not args.files:
+        parser.error("Either --all flag must be provided or at least one file must be specified")
+
+    if args.all and args.files:
+        parser.error("No files expected if using -a/--all flag")
+
+    analyze_and_update(**vars(args))
+
+
+if __name__ == '__main__':
+    cli()
diff --git a/scripts/create_denormalized_tables.py b/scripts/create_denormalized_tables.py
index 62a17083..84985ffc 100644
--- a/scripts/create_denormalized_tables.py
+++ b/scripts/create_denormalized_tables.py
@@ -31,74 +31,12 @@
 
 from typing import Any, Dict, List
 from synapseclient import Synapse, Column, Schema, Table
-from synapseclient.core.exceptions import SynapseAuthenticationError, SynapseNoCredentialsError
 import pandas as pd
 import numpy as np
 import re
-from scripts.utils import get_auth_token
-
-AUTH_TOKEN = get_auth_token()
-PROJECT_ID='syn63096806'
-
-# The synapse tables that hold source data
-SRC_TABLES = {      # getting rid of version numbers for now
-    'DataStandardOrTool': {        # convention in code, this abbreviation will be referred to as ..._tbl
-                    #   ..._table will usually refer to some kind of table object
-        'id': 'syn63096833', 'name': 'DataStandardOrTool',
-    },
-    'DataTopic': {
-        'id': 'syn63096835', 'name': 'DataTopic',
-    },
-    'Organization': {
-        'id': 'syn63096836', 'name': 'Organization',
-    },
-    'UseCase': { 'id': 'syn63096837', 'name': 'UseCase', },
-    'DataSubstrate': { 'id': 'syn63096834', 'name': 'DataSubstrate', }
-}
 
-# The table used for the explore landing page and to provide data for the home and detailed pages
-DEST_TABLES = {
-    'DST_denormalized': {
-        'dest_table_name': 'DST_denormalized',
-        'base_table': 'DataStandardOrTool',
-        'columns': [
-            {'faceted': False, 'name': 'id',                        'alias': 'id'},
-            {'faceted': False, 'name': 'name',                      'alias': 'acronym'},
-            {'faceted': False, 'name': 'description',               'alias': 'name'},
-            {'faceted': True,  'name': 'category',                  'alias': 'category', 'transform': 'camel_to_title_case'},
-            {'faceted': False, 'name': 'purpose_detail',            'alias': 'description'},
-            {'faceted': False, 'name': 'collection',                'alias': 'collections'},
-            {'faceted': False, 'name': 'concerns_data_topic',       'alias': 'concerns_data_topic'},
-            {'faceted': False, 'name': 'has_relevant_organization', 'alias': 'has_relevant_organization'},
-            {'faceted': False, 'name': 'responsible_organization',  'alias': 'responsible_organization'},
-            {'faceted': True,  'name': 'is_open',                   'alias': 'isOpen'},
-            {'faceted': True,  'name': 'requires_registration',     'alias': 'registration'},
-            {'faceted': False, 'name': 'url',                       'alias': 'URL'},
-            {'faceted': False, 'name': 'formal_specification',      'alias': 'formalSpec'},
-            {'faceted': False, 'name': 'publication',               'alias': 'publication'},
-            {'faceted': False, 'name': 'has_training_resource',     'alias': 'trainingResources'},
-            {'faceted': False, 'name': 'subclass_of',               'alias': 'subclassOf'},
-            {'faceted': False, 'name': 'contribution_date',         'alias': 'contributionDate'},
-            {'faceted': False, 'name': 'related_to',                'alias': 'relatedTo'},
-        ],
-        'join_columns': [
-            {'join_tbl': 'DataTopic', 'join_type': 'left', 'from': 'concerns_data_topic', 'to': 'id',
-             'dest_cols': [
-                {'faceted': True, 'name': 'name', 'alias': 'topic'},
-            ]},
-            {'join_tbl': 'Organization', 'join_type': 'left', 'from': 'has_relevant_organization', 'to': 'id',
-             'dest_cols': [
-                 {'faceted': True,  'name': 'name', 'alias': 'relevantOrgNames'},
-                 {'faceted': False, 'name': 'description', 'alias': 'relevantOrgDescriptions'},
-             ]},
-            {'join_tbl': 'Organization', 'join_type': 'left', 'from': 'responsible_organization', 'to': 'id',
-             'dest_cols': [
-                 {'faceted': False,  'name': 'name', 'alias': 'responsibleOrgAcronym'},
-                 {'faceted': False,  'name': 'description', 'alias': 'responsibleOrgName'},
-             ]},
-        ],
-    },
-}
+from scripts.generate_tables_config import DEST_TABLES, TABLE_IDS
+from scripts.utils import PROJECT_ID, create_or_clear_table, initialize_synapse
 
 TRANSFORMS = {
     # camel_to_title_case
@@ -113,16 +51,23 @@
 
 def denormalize_tables():
     syn = initialize_synapse()
-    src_tables = {tbl: get_src_table(syn, tbl) for tbl in SRC_TABLES}
+    src_tables = {}
 
     for dest_table in DEST_TABLES.values():
         base_tbl_name = dest_table['base_table']
-        base_df = src_tables[base_tbl_name]['df']
-
+        base_table_info = get_src_table(syn, TABLE_IDS[base_tbl_name])
+        base_df = base_table_info['df']
         if base_df.empty:
             print(f"Skipping '{dest_table['dest_table_name']}' — base table '{base_tbl_name}' has no data.")
             continue
 
+        # SRC_TABLE_NAMES is all source tables, src_table_names is just the ones for this dest table
+        src_table_names = set([base_tbl_name] + [j['join_tbl'] for j in dest_table['join_columns']])
+        for table_name in src_table_names:
+            table_info = TABLE_IDS[table_name]
+            table_info = get_src_table(syn, table_info)
+            src_tables[table_name] = table_info
+
         make_dest_table(syn, dest_table, src_tables)
 
 
@@ -229,24 +174,6 @@ def configure_column_metadata(columns: List[Dict[str, Any]], df: pd.DataFrame) -
 
         return columns
 
-    def create_or_clear_table(schema_name: str) -> None:
-        """
-        Delete all rows from a table if it already exists in Synapse. Takes a snapshot version for history.
-
-        :param schema_name: Name of the Synapse table to check and clear (if it already exists)
-        """
-        try:
-            existing_tables = syn.getChildren(PROJECT_ID, includeTypes=['table'])
-            for table in existing_tables:
-                if table['name'] == schema_name:
-                    query_result = syn.tableQuery(f"SELECT * FROM {table['id']}")
-                    syn.create_snapshot_version(table["id"])
-                    print(f"Table '{schema_name}' already exists. Deleting {len(query_result)} rows.")
-                    syn.delete(query_result)
-                    break
-        except Exception as e:
-            print(f"Error checking for existing table: {e}")
-
     # Step 1: Build all columns and their data
     base_columns = build_base_columns()
     join_columns = build_join_columns()
@@ -268,7 +195,7 @@ def create_or_clear_table(schema_name: str) -> None:
     )
 
     # Step 5: Clear existing table rows if applicable
-    create_or_clear_table(dest_table['dest_table_name'])
+    create_or_clear_table(syn, dest_table['dest_table_name'])
 
     # Step 6: Upload the table
     table = syn.store(Table(schema, final_df))
@@ -472,10 +399,13 @@ def create_json_column(
 
     return column_def
 
-
-def get_src_table(syn: Synapse, tbl: str) -> Dict[str, Any]:
+def get_src_table(syn: Synapse, table_info: Dict[str, Any]) -> Dict[str, Any]:
     """
-    Retrieve and validate a source table from Synapse, populate it with metadata and a cleaned DataFrame.
+    Retrieve and validate a source table from Synapse, capture its column metadata and a cleaned DataFrame.
+    If the table is empty:
+        - If a version number has been specified, raise an exception and quit execution.
+        - Otherwise, find the last non-empty version and use that
+            - Ideally,
 
     This function:
     - Retrieves the Synapse table and confirms its name matches the expected one
@@ -485,7 +415,9 @@ def get_src_table(syn: Synapse, tbl: str) -> Dict[str, Any]:
     - Stores the DataFrame, table object, and column info in the returned dictionary
 
     :param syn: Authenticated Synapse client
-    :param tbl: Key for the source table in the global SRC_TABLES dictionary
+    :param table_info: Dictionary with keys:
+             - 'id': Synapse table ID
+             - 'name': expected name of the table
     :return: Dictionary with keys:
              - 'id': Synapse table ID
              - 'name': expected name of the table
@@ -494,7 +426,6 @@ def get_src_table(syn: Synapse, tbl: str) -> Dict[str, Any]:
              - 'df': cleaned DataFrame of table contents
     :raises ValueError: if the retrieved Synapse table has a different name than expected
     """
-    table_info = SRC_TABLES[tbl]  # global lookup
     syn_table = syn.get(table_info['id'])
 
     # Confirm table name matches what's expected
@@ -518,16 +449,5 @@ def get_src_table(syn: Synapse, tbl: str) -> Dict[str, Any]:
     table_info['df'] = df
     return table_info
 
-def initialize_synapse() -> None:
-    """
-    Initialize the synapse client
-    """
-    try:
-        syn = Synapse()
-        syn.login(authToken=AUTH_TOKEN)
-        return syn
-    except (SynapseAuthenticationError, SynapseNoCredentialsError) as e:
-        raise Exception(f"Failed to authenticate with Synapse: {str(e)}")
-
 if __name__ == "__main__":
     denormalize_tables()
diff --git a/scripts/generate_tables_config.py b/scripts/generate_tables_config.py
new file mode 100644
index 00000000..ce25454c
--- /dev/null
+++ b/scripts/generate_tables_config.py
@@ -0,0 +1,143 @@
+# This configuration file is used to define the setup for the Synapse tables
+# and their relationships. It includes the source table names, destination table configurations,
+# and the Synapse table IDs, as well as how the tables will be rendered
+# in the Explorer (e.g., whether columns are faceted or not and how their
+# titles will be displayed).
+
+SRC_TABLE_NAMES = [
+    'Challenges',
+    'DataSet',
+    'DataStandardOrTool',
+    'DataSubstrate',
+    'DataTopic',
+    'Organization',
+    'UseCase',
+    # 'test',
+]
+TABLE_IDS = {
+    'Challenges': { 'name': 'Challenges', 'id': 'syn65913973' },
+    'CurrentTableVersions': { 'name': 'CurrentTableVersions', 'id': 'syn66330007' },
+    'DST_denormalized': { 'name': 'DST_denormalized', 'id': 'syn65676531' },
+    'DataSet': { 'name': 'DataSet', 'id': 'syn66330217' },
+    'DataStandardOrTool': { 'name': 'DataStandardOrTool', 'id': 'syn63096833' },
+    'DataSubstrate': { 'name': 'DataSubstrate', 'id': 'syn63096834' },
+    'DataTopic': { 'name': 'DataTopic', 'id': 'syn63096835' },
+    'Organization': { 'name': 'Organization', 'id': 'syn63096836' },
+    'UseCase': { 'name': 'UseCase', 'id': 'syn63096837' },
+    # 'test': { 'name': 'test', 'id': 'syn64943432' }
+}
+
+DEST_TABLES = {
+    # The table used for the explore landing page and to provide data for the home and detailed pages
+    'DST_denormalized': {
+        'dest_table_name': 'DST_denormalized',
+        'base_table': 'DataStandardOrTool',
+        'columns': [
+            {'faceted': False, 'name': 'id',                        'alias': 'id'},
+            {'faceted': False, 'name': 'name',                      'alias': 'acronym'},
+            {'faceted': False, 'name': 'description',               'alias': 'name'},
+            {'faceted': True,  'name': 'category',                  'alias': 'category', 'transform': 'camel_to_title_case'},
+            {'faceted': False, 'name': 'purpose_detail',            'alias': 'description'},
+            {'faceted': False, 'name': 'collection',                'alias': 'collections'},
+            {'faceted': False, 'name': 'concerns_data_topic',       'alias': 'concerns_data_topic'},
+            {'faceted': False, 'name': 'has_relevant_organization', 'alias': 'has_relevant_organization'},
+            {'faceted': False, 'name': 'responsible_organization',  'alias': 'responsible_organization'},
+            {'faceted': True,  'name': 'is_open',                   'alias': 'isOpen'},
+            {'faceted': True,  'name': 'requires_registration',     'alias': 'registration'},
+            {'faceted': False, 'name': 'url',                       'alias': 'URL'},
+            {'faceted': False, 'name': 'formal_specification',      'alias': 'formalSpec'},
+            {'faceted': False, 'name': 'publication',               'alias': 'publication'},
+            {'faceted': False, 'name': 'has_training_resource',     'alias': 'trainingResources'},
+            {'faceted': False, 'name': 'subclass_of',               'alias': 'subclassOf'},
+            {'faceted': False, 'name': 'contribution_date',         'alias': 'contributionDate'},
+            {'faceted': False, 'name': 'related_to',                'alias': 'relatedTo'},
+        ],
+        'join_columns': [
+            {'join_tbl': 'DataTopic', 'join_type': 'left', 'from': 'concerns_data_topic', 'to': 'id',
+             'dest_cols': [
+                {'faceted': True, 'name': 'name', 'alias': 'topic'},
+            ]},
+            {'join_tbl': 'Organization', 'join_type': 'left', 'from': 'has_relevant_organization', 'to': 'id',
+             'dest_cols': [
+                 {'faceted': True,  'name': 'name', 'alias': 'relevantOrgNames'},
+                 {'faceted': False, 'name': 'description', 'alias': 'relevantOrgDescriptions'},
+             ]},
+            {'join_tbl': 'Organization', 'join_type': 'left', 'from': 'responsible_organization', 'to': 'id',
+             'dest_cols': [
+                 {'faceted': False,  'name': 'name', 'alias': 'responsibleOrgAcronym'},
+                 {'faceted': False,  'name': 'description', 'alias': 'responsibleOrgName'},
+             ]},
+        ],
+    },
+    # 'DataSetPlus': {  # commented out while fixing other issues, but don't delete
+    #     'dest_table_name': 'DataSetPlus',
+    #     'base_table': 'Challenges',
+    #     'columns': [
+    #         {'faceted': False, 'name': 'headerImage', 'alias': 'imgId'},
+    #
+    #         """
+    #         dataset.name AS ds_name,  -- title
+    #         dataset.description AS ds_description,
+    #         # dataset.id,
+    #         # dataset.category,
+    #         dataset.data_url,
+    #         dataset.datasheet_url,
+    #         dataset.documentation_url,
+    #         dataset.has_files,
+    #         dataset.is_public,
+    #         # dataset.produced_by,
+    #         # dataset.substrates,
+    #         data
+    #         dataset.topics,
+    #         org.id,
+    #         org.category,
+    #         org.name,
+    #         org.description,
+    #         org.contributor_name,
+    #         org.contributor_github_name,
+    #         org.contributor_orcid,
+    #         org.ror_id,
+    #         org.wikidata_id,
+    #         org.url,
+    #         org.subclass_of,
+    #         challenges.headerImage,""",
+    #
+    #         {'faceted': False, 'name': 'id', 'alias': 'id'},
+    #         {'faceted': False, 'name': 'name', 'alias': 'acronym'},
+    #         {'faceted': False, 'name': 'description', 'alias': 'name'},
+    #         {'faceted': True, 'name': 'category', 'alias': 'category', 'transform': 'camel_to_title_case'},
+    #         {'faceted': False, 'name': 'purpose_detail', 'alias': 'description'},
+    #         {'faceted': False, 'name': 'collection', 'alias': 'collections'},
+    #         {'faceted': False, 'name': 'concerns_data_topic', 'alias': 'concerns_data_topic'},
+    #         {'faceted': False, 'name': 'has_relevant_organization', 'alias': 'has_relevant_organization'},
+    #         {'faceted': False, 'name': 'responsible_organization', 'alias': 'responsible_organization'},
+    #         {'faceted': True, 'name': 'is_open', 'alias': 'isOpen'},
+    #         {'faceted': True, 'name': 'requires_registration', 'alias': 'registration'},
+    #         {'faceted': False, 'name': 'url', 'alias': 'URL'},
+    #         {'faceted': False, 'name': 'formal_specification', 'alias': 'formalSpec'},
+    #         {'faceted': False, 'name': 'publication', 'alias': 'publication'},
+    #         {'faceted': False, 'name': 'has_training_resource', 'alias': 'trainingResources'},
+    #         {'faceted': False, 'name': 'subclass_of', 'alias': 'subclassOf'},
+    #         {'faceted': False, 'name': 'contribution_date', 'alias': 'contributionDate'},
+    #         {'faceted': False, 'name': 'related_to', 'alias': 'relatedTo'},
+    #     ],
+    #     'join_columns': [
+    #         {'join_tbl': 'DataTopic', 'join_type': 'left', 'from': 'concerns_data_topic', 'to': 'id',
+    #          'dest_cols': [
+    #              {'faceted': True, 'name': 'name', 'alias': 'topic'},
+    #          ]},
+    #         {'join_tbl': 'Organization', 'join_type': 'left', 'from': 'has_relevant_organization', 'to': 'id',
+    #          'dest_cols': [
+    #              {'faceted': True, 'name': 'name', 'alias': 'relevantOrgNames'},
+    #              {'faceted': False, 'name': 'description', 'alias': 'relevantOrgDescriptions'},
+    #          ]},
+    #         {'join_tbl': 'Organization', 'join_type': 'left', 'from': 'responsible_organization', 'to': 'id',
+    #          'dest_cols': [
+    #              {'faceted': False, 'name': 'name', 'alias': 'responsibleOrgAcronym'},
+    #              {'faceted': False, 'name': 'description', 'alias': 'responsibleOrgName'},
+    #          ]},
+    #     ],
+    # },
+}
+
+SRC_AND_DEST_TABLE_NAMES = SRC_TABLE_NAMES + list(DEST_TABLES.keys())
diff --git a/scripts/modify_synapse_schema.py b/scripts/modify_synapse_schema.py
deleted file mode 100644
index 0d0537d6..00000000
--- a/scripts/modify_synapse_schema.py
+++ /dev/null
@@ -1,371 +0,0 @@
-"""A python script for creating/updating B2AI standards explorer table schemas in synapse
-This script should be updated and rerun when the schema of the tables must be updated"""
-
-from synapseclient import Synapse, Table, Column, Schema
-from enum import Enum
-
-from scripts.utils import get_auth_token
-
-
-# Possible column names for the tables
-class ColumnName(Enum):
-    ID = "id"
-    CATEGORY = "category"
-    NAME = "name"
-    DESCRIPTION = "description"
-    CONTRIBUTOR_NAME = "contributor_name"
-    CONTRIBUTOR_GITHUB_NAME = "contributor_github_name"
-    CONTRIBUTOR_ORCID = "contributor_orcid"
-    COLLECTION = "collection"
-    CONCERNS_DATA_TOPIC = "concerns_data_topic"
-    PURPOSE_DETAIL = "purpose_detail"
-    SUBCLASS_OF = "subclass_of"
-    URL = "url"
-    IS_OPEN = "is_open"
-    REQUIRES_REGISTRATION = "requires_registration"
-    EDAM_ID = "edam_id"
-    MESH_ID = "mesh_id"
-    NCIT_ID = "ncit_id"
-    RELATED_TO = "related_to"
-    METADATA_STORAGE = "metadata_storage"
-    FILE_EXTENSIONS = "file_extensions"
-    LIMITATIONS = "limitations"
-    KNOWN_LIMITATIONS = "known_limitations"
-    CONTRIBUTION_DATE = "contribution_date"
-    PUBLICATION = "publication"
-    FORMAL_SPECIFICATION = "formal_specification"
-    HAS_RELEVANT_ORGANIZATION = "has_relevant_organization"
-    HAS_TRAINING_RESOURCE = "has_training_resource"
-    USE_CASE_CATEGORY = "use_case_category"
-    RELEVANCE_TO_DGPS = "relevance_to_dgps"
-    DATA_TOPICS = "data_topics"
-    STANDARDS_AND_TOOLS_FOR_DGP_USE = "standards_and_tools_for_dgp_use"
-    ENABLES = "enables"
-    INVOLVED_IN_EXPERIMENTAL_DESIGN = "involved_in_experimental_design"
-    INVOLVED_IN_METADATA_MANAGEMENT = "involved_in_metadata_management"
-    INVOLVED_IN_QUALITY_CONTROL = "involved_in_quality_control"
-    XREF = "xref"
-    ALTERNATIVE_STANDARDS_AND_TOOLS = "alternative_standards_and_tools"
-    ROR_ID = "ror_id"
-    WIKIDATA_ID = "wikidata_id"
-    RESPONSIBLE_ORGANIZATION = "responsible_organization"
-    DOCUMENTATION_URL = "documentation_url"
-    DATA_URL = "data_url"
-    DATASHEET_URL = "datasheet_url"
-    HAS_FILES = "has_files"
-    IS_PUBLIC = "is_public"
-    PRODUCED_BY = "produced_by"
-    SUBSTRATES = "substrates"
-    TOPICS = "topics"
-    USED_IN_BRIDGE2AI = "used_in_bridge2ai"
-
-
-# Possible columns in the standards data tables
-COLUMN_TEMPLATES = {
-    ColumnName.ID: Column(
-        name=ColumnName.ID.value, columnType="STRING", maximumSize=100
-    ),
-    ColumnName.CATEGORY: Column(
-        name=ColumnName.CATEGORY.value, columnType="STRING", maximumSize=100
-    ),
-    ColumnName.NAME: Column(name=ColumnName.NAME.value, columnType="MEDIUMTEXT"),
-    ColumnName.DESCRIPTION: Column(
-        name=ColumnName.DESCRIPTION.value, columnType="MEDIUMTEXT"
-    ),
-    ColumnName.CONTRIBUTOR_NAME: Column(
-        name=ColumnName.CONTRIBUTOR_NAME.value, columnType="STRING", maximumSize=100
-    ),
-    ColumnName.CONTRIBUTOR_GITHUB_NAME: Column(
-        name=ColumnName.CONTRIBUTOR_GITHUB_NAME.value,
-        columnType="STRING",
-        maximumSize=40,
-    ),
-    ColumnName.CONTRIBUTOR_ORCID: Column(
-        name=ColumnName.CONTRIBUTOR_ORCID.value, columnType="STRING", maximumSize=50
-    ),
-    ColumnName.COLLECTION: Column(
-        name="collection", columnType="STRING_LIST", maximumListLength=25
-    ),
-    ColumnName.PURPOSE_DETAIL: Column(name="purpose_detail", columnType="MEDIUMTEXT"),
-    ColumnName.CONCERNS_DATA_TOPIC: Column(
-        name="concerns_data_topic", columnType="STRING_LIST", maximumListLength=25
-    ),
-    ColumnName.SUBCLASS_OF: Column(
-        name=ColumnName.SUBCLASS_OF.value,
-        columnType="STRING_LIST",
-        maximumListLength=25,
-    ),
-    ColumnName.URL: Column(name=ColumnName.URL.value, columnType="MEDIUMTEXT"),
-    ColumnName.IS_OPEN: Column(name=ColumnName.IS_OPEN.value, columnType="BOOLEAN"),
-    ColumnName.REQUIRES_REGISTRATION: Column(
-        name=ColumnName.REQUIRES_REGISTRATION.value, columnType="BOOLEAN"
-    ),
-    ColumnName.EDAM_ID: Column(
-        name=ColumnName.EDAM_ID.value, columnType="STRING", maximumSize=25
-    ),
-    ColumnName.MESH_ID: Column(
-        name=ColumnName.MESH_ID.value, columnType="STRING", maximumSize=25
-    ),
-    ColumnName.NCIT_ID: Column(
-        name=ColumnName.NCIT_ID.value, columnType="STRING", maximumSize=25
-    ),
-    ColumnName.RELATED_TO: Column(
-        name=ColumnName.RELATED_TO.value, columnType="STRING_LIST", maximumListLength=25
-    ),
-    ColumnName.METADATA_STORAGE: Column(
-        name=ColumnName.METADATA_STORAGE.value,
-        columnType="STRING_LIST",
-        maximumListLength=25,
-    ),
-    ColumnName.FILE_EXTENSIONS: Column(
-        name=ColumnName.FILE_EXTENSIONS.value,
-        columnType="STRING_LIST",
-        maximumListLength=25,
-    ),
-    ColumnName.LIMITATIONS: Column(
-        name=ColumnName.LIMITATIONS.value, columnType="MEDIUMTEXT"
-    ),
-    ColumnName.KNOWN_LIMITATIONS: Column(
-        name=ColumnName.KNOWN_LIMITATIONS.value, columnType="MEDIUMTEXT"
-    ),
-    ColumnName.CONTRIBUTION_DATE: Column(
-        name=ColumnName.CONTRIBUTION_DATE.value, columnType="STRING", maximumSize=50
-    ),
-    ColumnName.PUBLICATION: Column(
-        name=ColumnName.PUBLICATION.value, columnType="STRING", maximumSize=100
-    ),
-    ColumnName.FORMAL_SPECIFICATION: Column(
-        name=ColumnName.FORMAL_SPECIFICATION.value, columnType="MEDIUMTEXT"
-    ),
-    ColumnName.HAS_RELEVANT_ORGANIZATION: Column(
-        name=ColumnName.HAS_RELEVANT_ORGANIZATION.value,
-        columnType="STRING_LIST",
-        maximumListLength=25,
-    ),
-    ColumnName.HAS_TRAINING_RESOURCE: Column(
-        name=ColumnName.HAS_TRAINING_RESOURCE.value,
-        columnType="STRING_LIST",
-        maximumListLength=25,
-    ),
-    ColumnName.USE_CASE_CATEGORY: Column(
-        name=ColumnName.USE_CASE_CATEGORY.value, columnType="STRING", maximumSize=100
-    ),
-    ColumnName.RELEVANCE_TO_DGPS: Column(
-        name=ColumnName.RELEVANCE_TO_DGPS.value,
-        columnType="STRING_LIST",
-        maximumListLength=25,
-    ),
-    ColumnName.DATA_TOPICS: Column(
-        name=ColumnName.DATA_TOPICS.value,
-        columnType="STRING_LIST",
-        maximumListLength=25,
-    ),
-    ColumnName.STANDARDS_AND_TOOLS_FOR_DGP_USE: Column(
-        name=ColumnName.STANDARDS_AND_TOOLS_FOR_DGP_USE.value,
-        columnType="STRING_LIST",
-        maximumListLength=25,
-    ),
-    ColumnName.ENABLES: Column(
-        name=ColumnName.ENABLES.value, columnType="STRING_LIST", maximumListLength=25
-    ),
-    ColumnName.INVOLVED_IN_EXPERIMENTAL_DESIGN: Column(
-        name=ColumnName.INVOLVED_IN_EXPERIMENTAL_DESIGN.value, columnType="BOOLEAN"
-    ),
-    ColumnName.INVOLVED_IN_METADATA_MANAGEMENT: Column(
-        name=ColumnName.INVOLVED_IN_METADATA_MANAGEMENT.value, columnType="BOOLEAN"
-    ),
-    ColumnName.INVOLVED_IN_QUALITY_CONTROL: Column(
-        name=ColumnName.INVOLVED_IN_QUALITY_CONTROL.value, columnType="BOOLEAN"
-    ),
-    ColumnName.XREF: Column(
-        name=ColumnName.XREF.value, columnType="STRING_LIST", maximumListLength=25
-    ),
-    ColumnName.ALTERNATIVE_STANDARDS_AND_TOOLS: Column(
-        name=ColumnName.ALTERNATIVE_STANDARDS_AND_TOOLS.value,
-        columnType="STRING_LIST",
-        maximumListLength=25,
-    ),
-    ColumnName.ROR_ID: Column(
-        name=ColumnName.ROR_ID.value, columnType="STRING", maximumSize=35
-    ),
-    ColumnName.WIKIDATA_ID: Column(
-        name=ColumnName.WIKIDATA_ID.value, columnType="STRING", maximumSize=25
-    ),
-    ColumnName.RESPONSIBLE_ORGANIZATION: Column(
-        name=ColumnName.RESPONSIBLE_ORGANIZATION.value, columnType="STRING_LIST", maximumListLength=25
-    ),
-    ColumnName.DATA_URL: Column(name=ColumnName.DATA_URL.value, columnType="MEDIUMTEXT"),
-    ColumnName.DATASHEET_URL: Column(name=ColumnName.DATASHEET_URL.value, columnType="MEDIUMTEXT"),
-    ColumnName.DOCUMENTATION_URL: Column(name=ColumnName.DOCUMENTATION_URL.value, columnType="MEDIUMTEXT"),
-    ColumnName.HAS_FILES: Column(
-        name=ColumnName.HAS_FILES.value, columnType="STRING_LIST", maximumListLength=25
-    ),
-    ColumnName.IS_PUBLIC: Column(
-        name=ColumnName.IS_PUBLIC.value, columnType="BOOLEAN"
-    ),
-    ColumnName.PRODUCED_BY: Column(
-        name=ColumnName.PRODUCED_BY.value, columnType="STRING_LIST", maximumListLength=25
-    ),
-    ColumnName.SUBSTRATES: Column(
-        name=ColumnName.SUBSTRATES.value, columnType="STRING_LIST", maximumListLength=25
-    ),
-    ColumnName.TOPICS: Column(
-        name="topics", columnType="STRING_LIST", maximumListLength=25
-    ),
-    ColumnName.USED_IN_BRIDGE2AI: Column(
-        name=ColumnName.USED_IN_BRIDGE2AI.value, columnType="BOOLEAN"
-    )
-}
-
-
-class TableSchema(Enum):
-    """An enum for the data tables, containing their ids in the synapse project and their schemas"""
-
-    DataSet = {
-        "id": "syn66330217",
-        "columns": [
-            ColumnName.ID,
-            ColumnName.CATEGORY,
-            ColumnName.DATA_URL,
-            ColumnName.DATASHEET_URL,
-            ColumnName.DESCRIPTION,
-            ColumnName.DOCUMENTATION_URL,
-            ColumnName.HAS_FILES,
-            ColumnName.IS_PUBLIC,
-            ColumnName.NAME,
-            ColumnName.PRODUCED_BY,
-            ColumnName.SUBSTRATES,
-            ColumnName.TOPICS,
-        ],
-    }
-    DataStandardOrTool = {
-        "id": "syn63096833",
-        "columns": [
-            ColumnName.ID,
-            ColumnName.CATEGORY,
-            ColumnName.NAME,
-            ColumnName.DESCRIPTION,
-            ColumnName.CONTRIBUTOR_NAME,
-            ColumnName.CONTRIBUTOR_GITHUB_NAME,
-            ColumnName.CONTRIBUTOR_ORCID,
-            ColumnName.COLLECTION,
-            ColumnName.CONCERNS_DATA_TOPIC,
-            ColumnName.PURPOSE_DETAIL,
-            ColumnName.IS_OPEN,
-            ColumnName.REQUIRES_REGISTRATION,
-            ColumnName.URL,
-            ColumnName.FORMAL_SPECIFICATION,
-            ColumnName.HAS_RELEVANT_ORGANIZATION,
-            ColumnName.PUBLICATION,
-            ColumnName.HAS_TRAINING_RESOURCE,
-            ColumnName.SUBCLASS_OF,
-            ColumnName.CONTRIBUTION_DATE,
-            ColumnName.RELATED_TO,
-            ColumnName.RESPONSIBLE_ORGANIZATION,
-            ColumnName.USED_IN_BRIDGE2AI,
-        ],
-    }
-    DataSubstrate = {
-        "id": "syn63096834",
-        "columns": [
-            ColumnName.ID,
-            ColumnName.CATEGORY,
-            ColumnName.NAME,
-            ColumnName.DESCRIPTION,
-            ColumnName.SUBCLASS_OF,
-            ColumnName.RELATED_TO,
-            ColumnName.CONTRIBUTOR_NAME,
-            ColumnName.CONTRIBUTOR_GITHUB_NAME,
-            ColumnName.CONTRIBUTOR_ORCID,
-            ColumnName.EDAM_ID,
-            ColumnName.NCIT_ID,
-            ColumnName.METADATA_STORAGE,
-            ColumnName.FILE_EXTENSIONS,
-            ColumnName.LIMITATIONS,
-            ColumnName.MESH_ID,
-            ColumnName.CONTRIBUTION_DATE,
-        ],
-    }
-    DataTopic = {
-        "id": "syn63096835",
-        "columns": [
-            ColumnName.ID,
-            ColumnName.CATEGORY,
-            ColumnName.NAME,
-            ColumnName.DESCRIPTION,
-            ColumnName.SUBCLASS_OF,
-            ColumnName.CONTRIBUTOR_NAME,
-            ColumnName.CONTRIBUTOR_GITHUB_NAME,
-            ColumnName.CONTRIBUTOR_ORCID,
-            ColumnName.EDAM_ID,
-            ColumnName.MESH_ID,
-            ColumnName.NCIT_ID,
-            ColumnName.RELATED_TO,
-        ],
-    }
-    Organization = {
-        "id": "syn63096836",
-        "columns": [
-            ColumnName.ID,
-            ColumnName.CATEGORY,
-            ColumnName.NAME,
-            ColumnName.DESCRIPTION,
-            ColumnName.CONTRIBUTOR_NAME,
-            ColumnName.CONTRIBUTOR_GITHUB_NAME,
-            ColumnName.CONTRIBUTOR_ORCID,
-            ColumnName.ROR_ID,
-            ColumnName.WIKIDATA_ID,
-            ColumnName.URL,
-            ColumnName.SUBCLASS_OF,
-        ],
-    }
-    UseCase = {
-        "id": "syn63096837",
-        "columns": [
-            ColumnName.ID,
-            ColumnName.CATEGORY,
-            ColumnName.NAME,
-            ColumnName.DESCRIPTION,
-            ColumnName.CONTRIBUTOR_NAME,
-            ColumnName.CONTRIBUTOR_GITHUB_NAME,
-            ColumnName.CONTRIBUTOR_ORCID,
-            ColumnName.USE_CASE_CATEGORY,
-            ColumnName.RELEVANCE_TO_DGPS,
-            ColumnName.DATA_TOPICS,
-            ColumnName.STANDARDS_AND_TOOLS_FOR_DGP_USE,
-            ColumnName.ENABLES,
-            ColumnName.INVOLVED_IN_EXPERIMENTAL_DESIGN,
-            ColumnName.INVOLVED_IN_METADATA_MANAGEMENT,
-            ColumnName.INVOLVED_IN_QUALITY_CONTROL,
-            ColumnName.XREF,
-            ColumnName.KNOWN_LIMITATIONS,
-            ColumnName.ALTERNATIVE_STANDARDS_AND_TOOLS,
-        ],
-    }
-
-
-def main():
-    auth_token = get_auth_token()
-
-    syn = Synapse()
-    syn.login(authToken=auth_token)
-
-    project = syn.get("syn63096806")
-
-    for table_schema in TableSchema:
-        table_id = table_schema.value["id"]
-
-        # take snapshot of tables before updating them
-        syn.create_snapshot_version(table_id)
-
-        column_keys = table_schema.value["columns"]
-        columns = [COLUMN_TEMPLATES[key] for key in column_keys]
-
-        schema = Schema(name=table_schema.name,
-                        columns=columns, parent=project)
-        table = Table(schema, values=[])
-        syn.store(table)
-
-
-if __name__ == "__main__":
-    main()
diff --git a/scripts/utils.py b/scripts/utils.py
index 85d5021b..d825bc77 100644
--- a/scripts/utils.py
+++ b/scripts/utils.py
@@ -1,9 +1,20 @@
 import os
+from synapseclient import Synapse
+from synapseclient.core.exceptions import SynapseAuthenticationError, SynapseNoCredentialsError
+"""
+Expected Environment:
+    - AUTH_TOKEN will be retrieved by scripts.utils.get_auth_token()
+      Instructions for setting up your auth token are documented in the README.
+"""
+PROJECT_ID='syn63096806'
 
 def get_auth_token():
     """
     Retrieves the Synapse authentication token from the user's ~/.synapseConfig file.
 
+    Some tests have started failing during GitHub actions. Going to try looking for token
+    in evironment variables as well.
+
     The config file must contain an authentication token. See the README for information on how to set this up.
 
     :return: The authentication token found in the config tile.
@@ -12,6 +23,11 @@ def get_auth_token():
         ValueError: If the 'authtoken' line is missing, malformed, or empty
         RuntimeError: If an unexpected error occurs while reading the file.
     """
+
+    token = os.getenv('SYNAPSE_AUTH_TOKEN') or os.getenv('AUTH_TOKEN') or os.getenv('auth_token')
+    if token:
+        return token
+
     auth_file = os.path.expanduser("~/.synapseConfig")
 
     try:
@@ -30,3 +46,37 @@ def get_auth_token():
         raise RuntimeError(f"An error occurred while reading the auth file: {e}")
 
     raise ValueError(f"'authtoken' not found in {auth_file}")
+
+def initialize_synapse() -> None:
+    """
+    Initialize the synapse client
+    """
+    try:
+        syn = Synapse()
+        syn.login(authToken=get_auth_token())
+        return syn
+    except (SynapseAuthenticationError, SynapseNoCredentialsError) as e:
+        raise Exception(f"Failed to authenticate with Synapse: {str(e)}")
+
+def copy_list_omit_property(list_of_dicts, property_to_omit):
+    return [{key: value for key, value in d.items() if key != property_to_omit}
+            for d in list_of_dicts]
+
+def create_or_clear_table(syn: Synapse, table_name: str) -> None:
+    """
+    Delete all rows from a table if it already exists in Synapse. Takes a snapshot version for history.
+
+    :param syn: Authenticated Synapse client
+    :param table_name: Name of the Synapse table to check and clear (if it already exists)
+    """
+    try:
+        existing_tables = syn.getChildren(PROJECT_ID, includeTypes=['table'])
+        for table in existing_tables:
+            if table['name'] == table_name:
+                query_result = syn.tableQuery(f"SELECT * FROM {table['id']}")
+                syn.create_snapshot_version(table["id"])
+                print(f"Table '{table_name}' already exists. Deleting {len(query_result)} rows.")
+                syn.delete(query_result)
+                break
+    except Exception as e:
+        print(f"Error checking for existing table: {e}")
diff --git a/tests/test_slot_registration.py b/tests/test_slot_registration.py
deleted file mode 100644
index 97ed7d7e..00000000
--- a/tests/test_slot_registration.py
+++ /dev/null
@@ -1,130 +0,0 @@
-from enum import Enum
-from pathlib import Path
-import unittest
-from unittest.mock import patch
-import yaml
-import scripts.modify_synapse_schema as modify_synapse_schema
-
-class TestSlotRegistration(unittest.TestCase):
-	DATA_DIRECTORY = "src/data"
-	MODIFY_SYNAPSE_SCHEMA_FILE = "scripts/modify_synapse_schema.py"
-
-
-	def _get_slots_by_table(self):
-		"""
-		Returns a dictionary of slots used by each table.
-			Keys = table names
-			Values = sets of slots used by each table
-		"""
-		slots_by_table = {}
-		for file_path in Path(TestSlotRegistration.DATA_DIRECTORY).iterdir():
-			if((not file_path.is_file()) or file_path.stem == "DataSet"):	# 'DataSet' is incomplete
-				continue
-
-			with file_path.open() as file:
-				file_contents = yaml.safe_load(file)
-			table_data = list(file_contents.values())[0]
-
-			table_slots = set()
-			table_name = file_path.stem
-
-			for entry in table_data:
-				entry_slots = entry.keys()
-				table_slots.update(entry_slots)
-
-			slots_by_table[table_name] = table_slots
-
-		return slots_by_table
-
-
-	def test_slot_registration(self):
-		"""
-		Verify that every slot in each table has been added to all column definitions in 'modify_snyapse_schema.py'.
-		This includes the enums `ColumnName` and `TableSchema`, plus the dictionary `COLUMN_TEMPLATES`
-		"""
-		ColumnName_values = [entry.value for entry in modify_synapse_schema.ColumnName]
-		COLUMN_TEMPLATE_keys = list(modify_synapse_schema.COLUMN_TEMPLATES.keys())
-
-		for table, slots in self._get_slots_by_table().items():
-			for slot in slots:
-				# All slots should be registered in these places
-				self.assertTrue(
-					slot in ColumnName_values,
-					f"slot '{slot}' not in '{TestSlotRegistration.MODIFY_SYNAPSE_SCHEMA_FILE}' > 'ColumnName'"
-				)
-				self.assertTrue(
-					modify_synapse_schema.ColumnName(slot) in COLUMN_TEMPLATE_keys,
-					f"slot '{slot}' not in '{TestSlotRegistration.MODIFY_SYNAPSE_SCHEMA_FILE}' > 'COLUMN_TEMPLATES'"
-				)
-
-				# Table-specific slot registration
-				self.assertTrue(
-					modify_synapse_schema.ColumnName(slot) in modify_synapse_schema.TableSchema[table].value["columns"],
-					f"slot '{slot}' not in '{TestSlotRegistration.MODIFY_SYNAPSE_SCHEMA_FILE}' > 'TableSchema[{table}]'"
-				)
-
-
-
-class MockColumnName(Enum):
-	TEST_SLOT = "test_slot"
-
-MOCK_COLUMN_TEMPLATES = {
-	MockColumnName.TEST_SLOT: "Definition of column"
-}
-
-class MockTableSchema(Enum):
-	DataStandardOrTool = {
-		"id": "syn63096833",
-		"columns": [
-			MockColumnName.TEST_SLOT
-		],
-	}
-
-class TestSlotRegistrationTests(unittest.TestCase):
-	"""Tests for TestSlotRegistration.test_slot_registration()"""
-
-	def setUp(self):
-		table_slots = {"DataStandardOrTool": ['test_slot']}
-		self.table_slots_patcher = patch.object(TestSlotRegistration, '_get_slots_by_table', return_value=table_slots)
-		self.mock_table_slots = self.table_slots_patcher.start()
-		self.addCleanup(self.table_slots_patcher.stop)
-
-
-	def _run_target_test(self) -> unittest.TestResult:
-		test_case = TestSlotRegistration(methodName='test_slot_registration')
-		result = unittest.TestResult()
-		test_case.run(result)
-		return result
-
-	def test_unregistered_slot(self):
-		"""INVALID - Slot not defined anywhere"""
-		result = self._run_target_test()
-		self.assertEqual(len(result.failures), 1)
-
-
-	@patch("scripts.modify_synapse_schema.ColumnName", new=MockColumnName)
-	def test_slot_not_in_column_templates_or_table_schema(self):
-		"""INVALID - Slot not defined in 'COLUMN_TEMPLATES' or 'TableSchema'"""
-		result = self._run_target_test()
-		self.assertEqual(len(result.failures), 1)
-
-
-	@patch("scripts.modify_synapse_schema.ColumnName", new=MockColumnName)
-	@patch("scripts.modify_synapse_schema.COLUMN_TEMPLATES", new=MOCK_COLUMN_TEMPLATES)
-	def test_slot_not_in_table_schema(self):
-		"""INVALID - Slot not defined in 'TableSchema'"""
-		result = self._run_target_test()
-		self.assertEqual(len(result.failures), 1)
-
-
-	@patch("scripts.modify_synapse_schema.ColumnName", new=MockColumnName)
-	@patch("scripts.modify_synapse_schema.COLUMN_TEMPLATES", new=MOCK_COLUMN_TEMPLATES)
-	@patch("scripts.modify_synapse_schema.TableSchema", new=MockTableSchema)
-	def test_slot_is_registered(self):
-		"""VALID - Slot defined everywhere"""
-		result = self._run_target_test()
-		self.assertTrue(result.wasSuccessful())
-
-
-if __name__ == "__main__":
-    unittest.main()


Reviews:
user: login: korikuzma
body: In general, I think commented out code should be deleted since it can be confusing to know why it was commented out and when we should add it back in. I usually just like to delete and make an issue if it is something that we will need for the future. 
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-12T11:53:05Z
user: login: katie-perry
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-14T14:21:59Z
user: login: katie-perry
body: If we no longer need modify_synapse_schema, I would go ahead and remove it and also remove the information about it in the README and remove the tests that reference it. 

I also would prefer we remove all commented out code for now (looks like there's a table commented out in the dst_denormalized script)
state: DISMISSED
author_association: COLLABORATOR
submitted_at: 2025-05-14T14:38:44Z
user: login: Sigfried
body: I left the commented out definition of DataSetPlus because I'm working on that next.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-14T17:03:37Z
user: login: katie-perry
state: DISMISSED
author_association: COLLABORATOR
submitted_at: 2025-05-14T17:29:43Z
user: login: katie-perry
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-14T17:36:37Z
user: login: caufieldjh
body: Thanks @Sigfried 
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-05-14T20:14:59Z
user: login: caufieldjh
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T19:19:06Z
user: login: caufieldjh
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T19:21:14Z

Review comments:
diff_hunk: @@ -518,6 +575,24 @@ def get_src_table(syn: Synapse, tbl: str) -> Dict[str, Any]:
     table_info['df'] = df
     return table_info
 
+def create_or_clear_table(syn, table_name: str) -> None:
+    """
+    Delete all rows from a table if it already exists in Synapse. Takes a snapshot version for history.
+
+    :param table_name: Name of the Synapse table to check and clear (if it already exists)
+    """
path: scripts/create_denormalized_tables.py
user: login: korikuzma
body: ```suggestion
def create_or_clear_table(syn: Synapse, table_name: str) -> None:
    """
    Delete all rows from a table if it already exists in Synapse. Takes a snapshot version for history.

    :param syn: Authenticated Synapse client
    :param table_name: Name of the Synapse table to check and clear (if it already exists)
    """
```
created_at: 2025-05-12T11:51:50Z
author_association: CONTRIBUTOR
path: scripts/modify_synapse_schema.py
user: login: katie-perry
body: do we need this at all anymore since the update script will attempt to do this for us?
created_at: 2025-05-14T14:21:58Z
author_association: COLLABORATOR
diff_hunk: @@ -44,6 +44,7 @@ The scripts in this folder are designed to:
 Each script is intended to be run individually. Here’s how to use them:
 
 ### Example Script: modify_synapse_schema.py
+#### No longer using modify_synapse_schema. Should change example to update_synapse_tables maybe
path: scripts/README.md
user: login: katie-perry
body: We should update the example to be create_denormalized_tables, probably. I'm okay with handling that in a new ticket, since it would be nice to push this through 
created_at: 2025-05-14T17:29:39Z
author_association: COLLABORATOR
diff_hunk: @@ -44,6 +44,7 @@ The scripts in this folder are designed to:
 Each script is intended to be run individually. Here’s how to use them:
 
 ### Example Script: modify_synapse_schema.py
+#### No longer using modify_synapse_schema. Should change example to update_synapse_tables maybe
path: scripts/README.md
user: login: katie-perry
body: https://github.com/bridge2ai/b2ai-standards-registry/issues/314
created_at: 2025-05-14T17:36:36Z
author_association: COLLABORATOR

PR comments:
user: login: Sigfried
created_at: 2025-05-13T20:46:19Z
body: @korikuzma or @katiestahl, I don't know why these tests are failing now but not before. It looks like the offending lines are unchanged:

        auth_file = os.path.expanduser("~/.synapseConfig")
        with open(auth_file, "r") as file:

The error shows that the action runner doesn't have a ~/.synapseConfig:

    FileNotFoundError: Authentication file not found: /home/runner/.synapseConfig

Should it? Also, with the latest code, we no longer need to run modify_synapse_schema, so maybe we should get rid of some tests anyway.

Any suggestions?
author_association: COLLABORATOR
user: login: Sigfried
created_at: 2025-05-14T20:35:52Z
body: Apologies to reviewers. I just pushed some more changes in response to @katiestahl's comments above.

Besides reviewing the code, it would be good to test it a bit. @caufieldjh (or someone), could you maybe add or change something in the schema of one of the files that gets uploaded to synapse and then check on synapse that it worked? And if nothing needs to change, just change something and change it back? Maybe DataSet or UseCase? I don't think it will currently affect the explorer if one of those was broken for a bit.
author_association: COLLABORATOR
user: login: caufieldjh
created_at: 2025-05-14T20:40:57Z
body: Oh right, thanks for the reminder - I was going to do that.
~~Will post the relevant commit from `standards-schemas` here once it's made~~, then update the schema and the UseCase data table in this branch.
(I'll actually just change both the schema and data here if that's alright)
author_association: CONTRIBUTOR
user: login: caufieldjh
created_at: 2025-05-14T21:00:00Z
body: OK, this appears to update properly:
```
$ poetry run python -m scripts.analyze_and_update_synapse_tables -t UseCase
Signing in...

UPGRADE AVAILABLE

A more recent version of the Synapse Client (4.8.0) is available. Your version (4.7.0) can be upgraded by typing:
    pip install --upgrade synapseclient

Python Synapse Client version 4.8.0 release notes

https://python-docs.synapse.org/news/

Welcome, Harry Caufield!

Creating snapshot and clearing table project/data/UseCase.json
Querying table/view: 'syn63096837' ... [--------------------]0.00%   0/100       
Querying table/view: 'syn63096837' ... [####################]100.00%   100/100   Done...
    
Downloading files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 33.7k/33.7k [00:00<00:00, 143kB/s, syn63096837][syn63096837]: Downloaded to /home/harry/.synapseCache/902/156358902/SYNAPSE_TABLE_QUERY_156358902.csv
Downloading files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 33.7k/33.7k [00:00<00:00, 142kB/s, syn63096837]
Table 'UseCase' already exists. Deleting 44 rows.
Uploading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 372/372 [00:00<00:00, 1.11kB/s, tmpbv5e09um.csv]
Applying an update to table: 'syn63096837' ... [--------------------]0.00%   0/100       
Applying an update to table: 'syn63096837' ... [####################]100.00%   100/100   Done...
    
Populating table for file: project/data/UseCase.json
Uploading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 31.6k/31.6k [00:00<00:00, 133kB/s, table.csv]
Applying an update to table: 'syn63096837' ... [--------------------]0.00%   0/100       
Applying an update to table: 'syn63096837' ... [####################]100.00%   100/100   Done...
    
Finished populating table
```
author_association: CONTRIBUTOR
user: login: Sigfried
created_at: 2025-05-14T22:05:51Z
body: Thanks, @caufieldjh.

It looks like you did the update from the command line. Did you also let it update using the GitHub actions? I want to make sure that still works. 
author_association: COLLABORATOR
user: login: caufieldjh
created_at: 2025-05-14T22:17:08Z
body: Will that (the GH action for updating the tables) work without merging this PR first?
author_association: CONTRIBUTOR
user: login: Sigfried
created_at: 2025-05-14T22:33:23Z
body: Good point. Probably not.
author_association: COLLABORATOR
user: login: Sigfried
created_at: 2025-05-16T13:39:01Z
body: This is ready for final review and merging. I've already run the code and the Synapse tables all seem to be good and up-to-date. On the front end, https://github.com/Sage-Bionetworks/synapse-web-monorepo/pull/1865 needs to be merged so the portal will use the up-to-date tables. (I don't know if there are significant differences between the up-to-date versions and the ones used in the portal now.)
author_association: COLLABORATOR
user: login: Sigfried
created_at: 2025-05-16T16:31:12Z
body: @caufieldjh , could you review and merge this?
author_association: COLLABORATOR