Repository: Sage-Bionetworks/GENIE-Sponsored-Projects
Pull request #151: [GEN-1859] Add toggle to use cpt_seq_date form derived variables for replacement
Author: login: rxu17
Created: 2025-04-09T20:57:50Z | Merged: 2025-04-10T18:04:37Z

# **Problem:**
We want to stop using `SEQ_YEAR` in main genie consortium clinical file to replace `CPT_SEQ_DATE` values in cbio files in BPC post-processing / release. This is because we are trying to align bpc post-processing. In bpc post-processing we generate cbio export files AND clinical files. The clinical files use `cpt_seq_date` from the derived variable files while the cbio files (currently) use `SEQ_YEAR` from the main genie files. This can cause discrepancies.

JIRA Ticket: [GEN-1859](https://sagebionetworks.jira.com/browse/GEN-1859)

# **Solution:**
We will use `cpt_seq_date` from the stats team provided derived variable file to replace the `CPT_SEQ_DATE` values in cbio files in BPC post-processing / release.

**Major Data Processing Changes**
- Added function [`get_derived_variable_file`](https://github.com/Sage-Bionetworks/GENIE-Sponsored-Projects/blob/b88c1f96195b2bd5ae3600d8e7bc14a5aa2ca637/geniesp/bpc_redcap_export_mapping.py#L484-L498) to retrieve the derived variable file.
I decided not to make this a class cached method property like `genie_clinicaldf` because it's only used for this cpt_seq_date replacement (in future if it's used throughout like `genie_clinicaldf`, that could be considered. 

- Added function [`replace_cpt_seq_date`](https://github.com/Sage-Bionetworks/GENIE-Sponsored-Projects/blob/b88c1f96195b2bd5ae3600d8e7bc14a5aa2ca637/geniesp/bpc_redcap_export_mapping.py#L600-L658) to replace the cpt_seq_date values in the input cbio files with one of the methods: `main_genie` or `derived_variable`. For specific decisions regarding how the merges and replacement are handled (mainly data reasons), see comments on JIRA ticket.

**Design Changes**
I added a toggle called [cpt_seq_date_replacement_type](https://github.com/Sage-Bionetworks/GENIE-Sponsored-Projects/blob/gen-1859-use-seq-date/geniesp/__main__.py#L78-L84) to our nextflow pipeline and the calling of the bpc script, so we can switch between the **new** method of using the derived variable file and the **old** method of using the main genie clinical file in case in the future, we want to swap back to using main genie.

Relevant code related to implementing **just** this toggle externally to link with nextflow. There's plenty of changes in the main script bpc_redcap_export_mapping.py:
- [in __main__.py](https://github.com/Sage-Bionetworks/GENIE-Sponsored-Projects/compare/gen-1859-use-seq-date?expand=1#diff-6da71cb519f211bcfbe82560c4c6e7923699e5eba5b6aa3039c013813f142d79)
- [in main.nf](https://github.com/Sage-Bionetworks/GENIE-Sponsored-Projects/compare/gen-1859-use-seq-date?expand=1#diff-6401496ba455b9488ffa902a6e4d7732b2c60ff2d77c5c3ef96b28a7ac7d3b28)
- [in nextflow_schema.json](https://github.com/Sage-Bionetworks/GENIE-Sponsored-Projects/compare/gen-1859-use-seq-date?expand=1#diff-4ec6c97fca07ea9b1ff300c0c4488d0f097ae32a78f0e0b01e00b9df0c346eb6)

# **Testing:**
- Unit tests added
- Tested on nextflow pipeline for both main genie and derived variable file methods for cohort BrCa with the default consortium release settings that are in the config which is 16.7 main genie consortium release
- Tested and investigated data locally and ensured that the results are as expected and any expected differences in the data between the runs are expected. See [investigation](https://sagebionetworks.jira.com/browse/GEN-1859?focusedCommentId=245521) for more details

[GEN-1859]: https://sagebionetworks.jira.com/browse/GEN-1859?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ


Diff:
diff --git a/README.md b/README.md
index f9f8268..c2cc4e7 100644
--- a/README.md
+++ b/README.md
@@ -54,26 +54,30 @@ positional arguments:
   release               Specify bpc release (e.g. 1.1-consortium)
 
 optional arguments:
-  -h, --help            Show this help message and exit
-  --upload              Upload files into Synapse BPC staging directory. Default: false
+  -h, --help                        Show this help message and exit
+  --upload                          Upload files into Synapse BPC staging directory. Default: false
   --cbioportal {synapseID}
-                        Optional parameter to specify cbioportal folder
-                        location
-  --production          Whether to run in production mode or not. Default: false
-  --use-grs             Whether to use grs or use dd as primary mapping. Default: false
+                                    Optional parameter to specify cbioportal folder
+                                    location
+  --production                      Whether to run in production mode or not. Default: false
+  --use-grs                         Whether to use grs or use dd as primary mapping. Default: false
+  --cpt_seq_date_replacement_type   The replacement data type to use for the cpt_seq_date value replacement.
+                                     Default: derived_variable, Options: ["derived_variable", "main_genie"]
 ```
 
 Example command line:
 
-This runs the release pipeline for BLADDER 1.1 in non-production mode (staging) with GRS enabled.
+This runs the release pipeline for BLADDER 1.1 in non-production mode (staging) with GRS enabled using
+main genie clinical data to do the cpt_seq_date replacement
 
 ```
-geniesp BLADDER 1.1-consortium --upload --use-grs
+geniesp BLADDER 1.1-consortium --upload --use-grs --cpt_seq_date_replacement_type main_genie
 ```
 
 Example command using docker:
 
-This runs the release pipeline for PANC 1.1 in non-production mode (staging).
+This runs the release pipeline for PANC 1.1 in non-production mode (staging) using derived variable data to
+do the cpt_seq_date replacement
 ```
 docker run --rm -e SYNAPSE_AUTH_TOKEN=$SYNAPSE_AUTH_TOKEN geniesp geniesp PANC 1.1-consortium --upload
 ```
diff --git a/geniesp/__main__.py b/geniesp/__main__.py
index dfec108..926a541 100644
--- a/geniesp/__main__.py
+++ b/geniesp/__main__.py
@@ -75,6 +75,13 @@ def main():
         action="store_true",
         help="Whether to use grs or use dd as primary mapping.",
     )
+    parser.add_argument(
+        "--cpt_seq_date_replacement_type",
+        type=str,
+        help="What data replacement type to use for cpt_seq_date",
+        default="derived_variable",
+        choices = ["main_genie", "derived_variable"]
+    )
     args = parser.parse_args()
 
     numeric_level = getattr(logging, args.log.upper(), None)
@@ -96,6 +103,7 @@ def main():
         upload=args.upload,
         production=args.production,
         use_grs=args.use_grs,
+        cpt_seq_date_replacement_type = args.cpt_seq_date_replacement_type,
     ).run()
 
 
diff --git a/geniesp/bpc_redcap_export_mapping.py b/geniesp/bpc_redcap_export_mapping.py
index 36473e1..0443f4e 100644
--- a/geniesp/bpc_redcap_export_mapping.py
+++ b/geniesp/bpc_redcap_export_mapping.py
@@ -481,6 +481,24 @@ def get_data_file_synapse_id_df(syn: Synapse, synid_table_files: str) -> pd.Data
     return data_tablesdf
 
 
+def get_derived_variable_file(syn: Synapse, derived_var_synid : str, cohort : str) -> pd.DataFrame:
+    """Gets the cancer panel test derived variable file used for
+        replacing the cpt_seq_date values in the input BPC clinical files
+
+    Args:
+        syn (Synapse): Synapse connection
+        derived_var_synid (str): synapse id for derived variable file
+        cohort (str): name of the cohort to filter on
+
+    Returns:
+        pd.DataFrame: derived variable file for the specific cohort
+    """
+    df = pd.read_csv(syn.get(derived_var_synid).path, low_memory = True)
+    df = df.query(f"cohort == '{cohort}'")
+    return df
+
+
+
 def create_release_folders(cohort: str) -> None:
     """Create local folders for release folders.
 
@@ -579,9 +597,74 @@ def check_oncotree_codes(
             f"There are invalid values in ONCOTREE_CODE column in the clinical df: {invalid_codes}")
 
 
+def replace_cpt_seq_date(
+    input_data : pd.DataFrame, 
+    replacement_data : pd.DataFrame,
+    cpt_seq_date_replacement_type : str
+    ) -> pd.DataFrame:
+    """ Replaces CPT_SEQ_DATE in input BPC file with specified replacement data's 
+        CPT_SEQ_DATE because the values in CPT_SEQ_DATE are incorrect.
+        
+        Current available cpt_seq_date_replacement_type values are ["main_genie", "derived_variable"]
+            - main_genie - uses the main genie consortium release's sample file and 
+            - derived_variable - uses the stats team's provided derived variable file
+                after BPC processing is complete and tables are provided
+
+    Args:
+        input_data (pd.DataFrame): input data with CPT_SEQ_DATE values to be replaced
+        replacement_data (pd.DataFrame): the derived variable data with CPT_SEQ_DATE values to 
+            use as replacement
+        cpt_seq_date_replacement_type (str): the type of replacement used, (e.g: main genie, derived variable)
+    
+    Raises:
+        ValueError: thrown when cpt_seq_date_replacement_type is not one of the valid
+            allowed values
+    
+    Returns:
+        pd.DataFrame: input data with replaced CPT_SEQ_DATE values
+    """
+    # Remove CPT_SEQ_DATE because the values are incorrect
+    del input_data["CPT_SEQ_DATE"]
+    logging.info(f"Replacing CPT_SEQ_DATE values with {cpt_seq_date_replacement_type} values ...")
+    if cpt_seq_date_replacement_type == "main_genie":
+        
+        # main genie clinical is just sample data
+        replacement_data = replacement_data[["SAMPLE_ID", "SEQ_YEAR"]]
+        replacement_data.rename(columns={"SEQ_YEAR": "CPT_SEQ_DATE"}, inplace=True)
+        merge_cols = ["SAMPLE_ID"]
+        
+    elif cpt_seq_date_replacement_type == "derived_variable":
+        # there should only be a unique seq_date per sample id as 
+        # cpt_seq_date is unique field to clinical sample data
+        replacement_data = replacement_data[
+            ["cpt_genie_sample_id", "cpt_seq_date"]
+        ].drop_duplicates()
+        
+        # rename to match input data
+        replacement_data.rename(columns = {
+            "cpt_genie_sample_id" : "SAMPLE_ID", 
+            "cpt_seq_date" : "CPT_SEQ_DATE"
+            }, inplace = True)
+        merge_cols = ["SAMPLE_ID"]
+    else:
+        raise ValueError(f"cpt_seq_date_replacement_type: {cpt_seq_date_replacement_type} invalid!")
+
+    # Replace with replacement data's seq date variable
+    input_data = input_data.merge(
+        replacement_data,
+        on = merge_cols,
+        how="left",
+    )
+    return input_data
+
+
 class BpcProjectRunner(metaclass=ABCMeta):
     """BPC redcap to cbioportal export"""
     
+    # synapse_id of derived variable file with cpt_seq_date
+    # to use as replacement for CPT_SEQ_DATE in input cbio files
+    # instead of using main genie clinical file
+    _DERIVED_VARIABLE_SYNID = "syn22296823"
     _STAGING_RELEASES_FOLDER = {
         "production": "syn50876969",
         "staging":"syn64018253"
@@ -603,7 +686,7 @@ class BpcProjectRunner(metaclass=ABCMeta):
         "staging": "syn64018293"
     }
     # main GENIE release folder
-    # NOTE: Must use consortium release, because SEQ_YEAR is used
+    # NOTE: Must use consortium release, because SEQ_DATE is used
     # NOTE: Must match release tracking sheet and release table info
     # for the given cohort
     _MG_RELEASE_SYNID = "syn63602196"
@@ -629,7 +712,16 @@ class BpcProjectRunner(metaclass=ABCMeta):
     # cohort-generic link to documentation for cBio files
     _url_cbio = "https://docs.google.com/document/d/1IBVF-FLecUG8Od6mSEhYfWH3wATLNMnZcBw2_G0jSAo/edit"
 
-    def __init__(self, syn, cbiopath, release, upload=False, production=False, use_grs=False):
+    def __init__(
+        self, 
+        syn, 
+        cbiopath, 
+        release, 
+        upload=False, 
+        production=False, 
+        use_grs=False, 
+        cpt_seq_date_replacement_type = "derived_variable"
+        ):
         if not os.path.exists(cbiopath):
             raise ValueError("cbiopath doesn't exist")
         if self._SPONSORED_PROJECT == "":
@@ -642,6 +734,7 @@ def __init__(self, syn, cbiopath, release, upload=False, production=False, use_g
         self.production = production
         self.environment = "production" if self.production else "staging"
         self.use_grs = use_grs
+        self.cpt_seq_date_replacement_type = cpt_seq_date_replacement_type
 
     @cached_property
     def genie_clinicaldf(self) -> pd.DataFrame:
@@ -663,7 +756,7 @@ def genie_clinicaldf(self) -> pd.DataFrame:
         # This is due to the most recent releases potentially having
         # samples retracted. The consortium release matched with the
         # public release (14.7-consortium <-> 14.0-public) must be used
-        # due to SEQ_YEAR being used through the code.
+        # due to SEQ_DATE being used through the code.
         sample_synid = self.get_mg_synid(
             self._MG_RELEASE_SYNID, "data_clinical_sample.txt"
         )
@@ -1867,11 +1960,20 @@ def get_patient(self, df_map: pd.DataFrame, df_file: pd.DataFrame) -> pd.DataFra
         return df_patient_subset[cols_to_order]
 
     def get_sample(self, df_map: pd.DataFrame, df_file: pd.DataFrame) -> pd.DataFrame:
-        """SAMPLE data file
+        """Gets the SAMPLE clinical data file and runs through some
+            transformations, notably:
+                - subsets input on main genie samples
+                - converts variables(days) to variables(years)
+                - replaces CPT_SEQ_DATE values
+                - sorts on PDL1_POSITIVE_ANY values
 
         Args:
             df_map (pd.DataFrame): variable to cBioPortal mapping info
             df_file (pd.DataFrame): data file to Synapse ID mapping
+            
+        Raises:
+            ValueError: thrown when cpt_seq_date_replacement_type is not one of the valid
+            allowed values
 
         Returns:
             pd.DataFrame: SAMPLE data
@@ -1906,20 +2008,33 @@ def get_sample(self, df_map: pd.DataFrame, df_file: pd.DataFrame) -> pd.DataFram
         df_sample_subset["AGE_AT_SEQUENCING"] = df_sample_subset[
             "AGE_AT_SEQUENCING"
         ].apply(np.floor)
-        # Remove CPT_SEQ_DATE because the values are incorrect
-        del df_sample_subset["CPT_SEQ_DATE"]
-        # Obtain this information from the main GENIE cohort
-        df_sample_subset = df_sample_subset.merge(
-            self.genie_clinicaldf[["SAMPLE_ID", "SEQ_YEAR"]],
-            on="SAMPLE_ID",
-            how="left",
-        )
-        df_sample_subset.rename(columns={"SEQ_YEAR": "CPT_SEQ_DATE"}, inplace=True)
+        
+        if self.cpt_seq_date_replacement_type == "derived_variable":
+            derived_df = get_derived_variable_file(
+                syn = self.syn, 
+                derived_var_synid = self._DERIVED_VARIABLE_SYNID, 
+                cohort = self._SPONSORED_PROJECT
+            )
+            df_sample_subset = replace_cpt_seq_date(
+                input_data = df_sample_subset, 
+                replacement_data = derived_df,
+                cpt_seq_date_replacement_type = self.cpt_seq_date_replacement_type
+            )
+        elif self.cpt_seq_date_replacement_type == "main_genie":
+            df_sample_subset = replace_cpt_seq_date(
+                input_data = df_sample_subset, 
+                replacement_data = self.genie_clinicaldf,
+                cpt_seq_date_replacement_type = self.cpt_seq_date_replacement_type
+            )
+        else:
+            raise ValueError(f"cpt_seq_date_replacement_type: {self.cpt_seq_date_replacement_type} invalid!")
+            
         df_sample_subset.sort_values("PDL1_POSITIVE_ANY", ascending=False, inplace=True)
         df_sample_subset.drop_duplicates("SAMPLE_ID", inplace=True)
 
         return df_sample_subset
-    
+        
+
     def create_and_write_case_lists(
         self, subset_sampledf: pd.DataFrame, subset_patientdf: pd.DataFrame, used: list
     ) -> None:
diff --git a/main.nf b/main.nf
index 1df25b0..aca9b3d 100644
--- a/main.nf
+++ b/main.nf
@@ -13,6 +13,7 @@ process cBioPortalExport {
    val release
    val production
    val use_grs
+   val cpt_seq_date_replacement_type
 
    output:
    stdout
@@ -24,27 +25,31 @@ process cBioPortalExport {
          --upload \
          --cbioportal /usr/src/cbioportal \
          --production \
-         --use-grs
+         --use-grs \
+         --cpt_seq_date_replacement_type $cpt_seq_date_replacement_type
       """
    } else if (production && !use_grs){
       """
       geniesp $cohort $release \
          --upload \
          --cbioportal /usr/src/cbioportal \
-         --production
+         --production \
+         --cpt_seq_date_replacement_type $cpt_seq_date_replacement_type
       """
    } else if (!production && use_grs){
       """
       geniesp $cohort $release \
          --upload \
          --cbioportal /usr/src/cbioportal \
-         --use-grs
+         --use-grs \
+         --cpt_seq_date_replacement_type $cpt_seq_date_replacement_type
       """
    } else {
       """
       geniesp $cohort $release \
          --upload \
          --cbioportal /usr/src/cbioportal \
+         --cpt_seq_date_replacement_type $cpt_seq_date_replacement_type
       """
    }
 }
@@ -54,6 +59,7 @@ workflow {
    params.release = '1.1-consortium'  // Default
    params.production = false
    params.use_grs = false
+   params.cpt_seq_date_replacement_type = 'derived_variable'
 
    // Check if cohort is part of allowed cohort list
    def allowed_cohorts = ["BLADDER", "BrCa", "CRC", "ESOPHAGO", "MELANOMA", "NSCLC", "OVARIAN", "PANC", "Prostate", "RENAL"]
@@ -63,6 +69,7 @@ workflow {
    ch_release = Channel.value(params.release)
    ch_production = Channel.value(params.production)
    ch_use_grs = Channel.value(params.use_grs)
+   ch_cpt_seq_date_replacement_type = Channel.value(params.cpt_seq_date_replacement_type)
 
-   cBioPortalExport(ch_cohort, ch_release, ch_production, ch_use_grs)
+   cBioPortalExport(ch_cohort, ch_release, ch_production, ch_use_grs, ch_cpt_seq_date_replacement_type)
 }
diff --git a/nextflow_schema.json b/nextflow_schema.json
index 98b16a8..0692487 100644
--- a/nextflow_schema.json
+++ b/nextflow_schema.json
@@ -53,6 +53,15 @@
                         false
                     ]
                 },
+                "cpt_seq_date_replacement_type": {
+                    "type": "string",
+                    "default": "derived_variable",
+                    "description": "Replacement data type to be used for cpt_seq_date.",
+                    "enum": [
+                        "derived_variable",
+                        "main_genie"
+                    ]
+                },
                 "geniesp_docker":{
                     "type": "string",
                     "description": "Name of docker to use for release process in geniesp"
diff --git a/tests/test_bpc_redcap_export_mapping.py b/tests/test_bpc_redcap_export_mapping.py
index 09da045..8236e34 100644
--- a/tests/test_bpc_redcap_export_mapping.py
+++ b/tests/test_bpc_redcap_export_mapping.py
@@ -3,6 +3,7 @@
 from unittest import mock
 
 import pandas as pd
+from pandas.testing import assert_frame_equal
 import synapseclient
 
 from geniesp import bpc_redcap_export_mapping as bpc_export
@@ -131,7 +132,7 @@ def test_that_parse_drug_mappings(input_mapping, var_names, output_mapping):
                 "RCC": {"CANCER_TYPE": "Renal Cell Carcinoma"},
                 "OVARY": {"CANCER_TYPE": "Ovarian Cancer"},
             },
-            "There are invalid values in ONCOTREE_CODE column in the clinical df: ['Renal Cell Carcinoma', 'Renal Clear Cell Carcinoma']",
+            "There are invalid values in ONCOTREE_CODE column in the clinical df: ['Renal Clear Cell Carcinoma', 'Renal Cell Carcinoma']",
         ),
         (
             pd.DataFrame(dict(ONCOTREE_CODE=["Renal Cell Carcinoma", "RCC"])),
@@ -164,3 +165,250 @@ def test_that_check_oncotree_codes_gives_no_warning_when_all_codes_valid(caplog)
         "There are invalid values in ONCOTREE_CODE column in the clinical df"
         not in caplog.text
     )
+
+
+def test_that_get_derived_variable_file_gets_file_correctly(mock_syn):
+    test_df = pd.DataFrame(
+        dict(
+            cohort = ["BLADDER", "BrCa", "BLADDER"],
+            record_id = ["GENIE-SAGE-1", "GENIE-SAGE-2", "GENIE-SAGE-3"]
+            )
+    )
+    with mock.patch.object(mock_syn, "get") as mock_syn_get, mock.patch.object(
+        pd, "read_csv", return_value = test_df
+        ) as mock_read_csv:
+            output = bpc_export.get_derived_variable_file(
+                mock_syn, 
+                derived_var_synid = "synZZZZ", 
+                cohort = "BLADDER"
+                )
+            assert_frame_equal(
+                output.reset_index(drop=True), pd.DataFrame(
+                    dict(
+                        cohort = ["BLADDER", "BLADDER"],
+                        record_id = ["GENIE-SAGE-1", "GENIE-SAGE-3"]
+                        )
+                ).reset_index(drop=True),
+                check_index_type=False
+            )
+    
+
+@pytest.mark.parametrize(
+    "input, clinical, expected",
+    [
+        (pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "record_id": ["GENIE-1", "GENIE-1"],
+                    "cpt_genie_sample_id": ["GENIE-1-3", "GENIE-1-4"],
+                    "cpt_seq_date": ["2017", "2018"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": [None, None],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "record_id": ["GENIE-1", "GENIE-1"],
+                    "cpt_genie_sample_id": ["GENIE-1-1", "GENIE-1-3"],
+                    "cpt_seq_date": ["2017", "2018"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2017", None],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "record_id": ["GENIE-1", "GENIE-1"],
+                    "cpt_genie_sample_id": ["GENIE-1-1", "GENIE-1-2"],
+                    "cpt_seq_date": ["2017", "2018"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2017", "2018"],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "record_id": ["GENIE-1", "GENIE-1"],
+                    "cpt_genie_sample_id": ["GENIE-1-1", "GENIE-1-2"],
+                    "cpt_seq_date": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2012", "2013"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "record_id": ["GENIE-1", "GENIE-1", "GENIE-1"],
+                    "cpt_genie_sample_id": ["GENIE-1-1", "GENIE-1-2", "GENIE-1-2"],
+                    "cpt_seq_date": ["2014", "2015", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ))
+        ],
+    ids = [
+        "none_replaced", 
+        "some_replaced", 
+        "all_replaced", 
+        "the_same", 
+        "replacement_has_dups"
+        ]
+)
+def test_that_replace_cpt_seq_date_replaces_correctly_with_derived_variable_replacement_type(input, clinical, expected):
+    output = bpc_export.replace_cpt_seq_date(
+        input_data = input, 
+        replacement_data= clinical,
+        cpt_seq_date_replacement_type = "derived_variable"
+        )
+    assert_frame_equal(output, expected)
+
+
+@pytest.mark.parametrize(
+    "input, clinical, expected",
+    [
+        (pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-3", "GENIE-1-4"],
+                    "SEQ_YEAR": ["2017", "2018"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": [None, None],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-3"],
+                    "SEQ_YEAR": ["2017", "2018"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2017", None],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "SEQ_YEAR": ["2017", "2018"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2017", "2018"],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "SEQ_YEAR": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ))
+        ],
+    ids = ["none_replaced_with_extra_cols", "some_replaced", "all_replaced", "the_same"]
+)
+def test_that_replace_cpt_seq_date_replaces_correctly_with_main_genie_replacement_type(input, clinical, expected):
+    output = bpc_export.replace_cpt_seq_date(
+        input_data = input, 
+        replacement_data= clinical, 
+        cpt_seq_date_replacement_type = "main_genie"
+        )
+    assert_frame_equal(output, expected)
+    
+    
+def test_that_replace_cpt_seq_date_raises_value_error():
+    with pytest.raises(
+        ValueError, 
+        match = "cpt_seq_date_replacement_type: invalid_cpt_seq_date_replacement_type invalid!"
+        ):
+        output = bpc_export.replace_cpt_seq_date(
+            input_data = pd.DataFrame(
+                    {
+                        "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                        "CPT_SEQ_DATE": ["2014", "2015"],
+                    }
+                ), 
+            replacement_data= pd.DataFrame(), 
+            cpt_seq_date_replacement_type = "invalid_cpt_seq_date_replacement_type"
+            )


Reviews:
user: login: rxu17
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T21:17:24Z
user: login: rxu17
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T21:18:26Z
user: login: danlu1
body: LGTM, just a few comments. 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T23:34:21Z
user: login: rxu17
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T23:41:06Z
user: login: rxu17
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T23:42:42Z
user: login: thomasyu888
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-09T23:57:29Z
user: login: thomasyu888
body: 🔥 LGTM! Ill leave it to Dan for final review!
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-10T00:02:07Z
user: login: danlu1
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T00:02:08Z
user: login: rxu17
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T00:04:49Z
user: login: rxu17
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T00:05:49Z
user: login: rxu17
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-10T00:12:07Z

Review comments:
diff_hunk: @@ -481,6 +481,24 @@ def get_data_file_synapse_id_df(syn: Synapse, synid_table_files: str) -> pd.Data
     return data_tablesdf
 
 
+def get_derived_variable_file(syn: Synapse, derived_var_synid : str, cohort : str) -> pd.DataFrame:
path: geniesp/bpc_redcap_export_mapping.py
user: login: rxu17
body: Note: Because the stats team QCs the derived variable files, we will not be adding any in code QC/checks for it (assume no duplicates, issues with the actual values, etc
created_at: 2025-04-09T21:17:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -579,9 +597,74 @@ def check_oncotree_codes(
             f"There are invalid values in ONCOTREE_CODE column in the clinical df: {invalid_codes}")
 
 
+def replace_cpt_seq_date(
+    input_data : pd.DataFrame, 
+    replacement_data : pd.DataFrame,
+    cpt_seq_date_replacement_type : str
+    ) -> pd.DataFrame:
+    """ Replaces CPT_SEQ_DATE in input BPC file with specified replacement data's 
+        CPT_SEQ_DATE because the values in CPT_SEQ_DATE are incorrect.
+        
+        Current available cpt_seq_date_replacement_type values are ["main_genie", "derived_variable"]
+            - main_genie - uses the main genie consortium release's sample file and 
+            - derived_variable - uses the stats team's provided derived variable file
+                after BPC processing is complete and tables are provided
+
+    Args:
+        input_data (pd.DataFrame): input data with CPT_SEQ_DATE values to be replaced
+        replacement_data (pd.DataFrame): the derived variable data with CPT_SEQ_DATE values to 
+            use as replacement
+        cpt_seq_date_replacement_type (str): the type of replacement used, (e.g: main genie, derived variable)
+    
+    Raises:
+        ValueError: thrown when cpt_seq_date_replacement_type is not one of the valid
+            allowed values
+    
+    Returns:
+        pd.DataFrame: input data with replaced CPT_SEQ_DATE values
+    """
+    # Remove CPT_SEQ_DATE because the values are incorrect
+    del input_data["CPT_SEQ_DATE"]
+    logging.info(f"Replacing CPT_SEQ_DATE values with {cpt_seq_date_replacement_type} values ...")
+    if cpt_seq_date_replacement_type == "main_genie":
path: geniesp/bpc_redcap_export_mapping.py
user: login: rxu17
body: Note: This is the original replacement method. I kept the code logic the same
created_at: 2025-04-09T21:18:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -54,26 +54,30 @@ positional arguments:
   release               Specify bpc release (e.g. 1.1-consortium)
 
 optional arguments:
-  -h, --help            Show this help message and exit
-  --upload              Upload files into Synapse BPC staging directory. Default: false
+  -h, --help                        Show this help message and exit
+  --upload                          Upload files into Synapse BPC staging directory. Default: false
   --cbioportal {synapseID}
-                        Optional parameter to specify cbioportal folder
-                        location
-  --production          Whether to run in production mode or not. Default: false
-  --use-grs             Whether to use grs or use dd as primary mapping. Default: false
+                                    Optional parameter to specify cbioportal folder
+                                    location
+  --production                      Whether to run in production mode or not. Default: false
+  --use-grs                         Whether to use grs or use dd as primary mapping. Default: false
+  --cpt_seq_date_replacement_type   The replacement data type to use for the cpt_seq_date value replacement.
+                                     Default: derived_variable
path: README.md
user: login: danlu1
body: nit: Add description for all valid options for better clarity.
created_at: 2025-04-09T22:47:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -54,26 +54,30 @@ positional arguments:
   release               Specify bpc release (e.g. 1.1-consortium)
 
 optional arguments:
-  -h, --help            Show this help message and exit
-  --upload              Upload files into Synapse BPC staging directory. Default: false
+  -h, --help                        Show this help message and exit
+  --upload                          Upload files into Synapse BPC staging directory. Default: false
   --cbioportal {synapseID}
-                        Optional parameter to specify cbioportal folder
-                        location
-  --production          Whether to run in production mode or not. Default: false
-  --use-grs             Whether to use grs or use dd as primary mapping. Default: false
+                                    Optional parameter to specify cbioportal folder
+                                    location
+  --production                      Whether to run in production mode or not. Default: false
+  --use-grs                         Whether to use grs or use dd as primary mapping. Default: false
+  --cpt_seq_date_replacement_type   The replacement data type to use for the cpt_seq_date value replacement.
+                                     Default: derived_variable
 ```
 
 Example command line:
 
-This runs the release pipeline for BLADDER 1.1 in non-production mode (staging) with GRS enabled.
+This runs the release pipeline for BLADDER 1.1 in non-production mode (staging) with GRS enabled using
+main genie clinical data to do the cpt_seq_date replacement
path: README.md
user: login: danlu1
body: nit: this is also a example to use grs as primary mapping
created_at: 2025-04-09T22:50:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -579,9 +597,74 @@ def check_oncotree_codes(
             f"There are invalid values in ONCOTREE_CODE column in the clinical df: {invalid_codes}")
 
 
+def replace_cpt_seq_date(
+    input_data : pd.DataFrame, 
+    replacement_data : pd.DataFrame,
+    cpt_seq_date_replacement_type : str
+    ) -> pd.DataFrame:
+    """ Replaces CPT_SEQ_DATE in input BPC file with specified replacement data's 
+        CPT_SEQ_DATE because the values in CPT_SEQ_DATE are incorrect.
+        
+        Current available cpt_seq_date_replacement_type values are ["main_genie", "derived_variable"]
+            - main_genie - uses the main genie consortium release's sample file and 
+            - derived_variable - uses the stats team's provided derived variable file
+                after BPC processing is complete and tables are provided
+
+    Args:
+        input_data (pd.DataFrame): input data with CPT_SEQ_DATE values to be replaced
+        replacement_data (pd.DataFrame): the derived variable data with CPT_SEQ_DATE values to 
+            use as replacement
+        cpt_seq_date_replacement_type (str): the type of replacement used, (e.g: main genie, derived variable)
+    
+    Raises:
+        ValueError: thrown when cpt_seq_date_replacement_type is not one of the valid
+            allowed values
+    
+    Returns:
+        pd.DataFrame: input data with replaced CPT_SEQ_DATE values
+    """
+    # Remove CPT_SEQ_DATE because the values are incorrect
+    del input_data["CPT_SEQ_DATE"]
+    logging.info(f"Replacing CPT_SEQ_DATE values with {cpt_seq_date_replacement_type} values ...")
+    if cpt_seq_date_replacement_type == "main_genie":
+        
+        # main genie clinical is just sample data
+        replacement_data = replacement_data[["SAMPLE_ID", "SEQ_YEAR"]]
+        replacement_data.rename(columns={"SEQ_YEAR": "CPT_SEQ_DATE"}, inplace=True)
+        merge_cols = ["SAMPLE_ID"]
+        
+    elif cpt_seq_date_replacement_type == "derived_variable":
+        # there should only be a unique seq_date per patient id and sample id
+        replacement_data = replacement_data[
+            ["cpt_genie_sample_id", "record_id", "cpt_seq_date"]
path: geniesp/bpc_redcap_export_mapping.py
user: login: danlu1
body: Nit. As the investigation attached below, I believe that the `cpt_genie_sample_id` should be good enough to be the unique identifiers used for merging. 

<img width="1074" alt="Screenshot 2025-04-09 at 4 28 01 PM" src="https://github.com/user-attachments/assets/6e9439d8-44e5-4aa6-93dc-dce2ca49e3e5" />

created_at: 2025-04-09T23:28:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -54,26 +54,30 @@ positional arguments:
   release               Specify bpc release (e.g. 1.1-consortium)
 
 optional arguments:
-  -h, --help            Show this help message and exit
-  --upload              Upload files into Synapse BPC staging directory. Default: false
+  -h, --help                        Show this help message and exit
+  --upload                          Upload files into Synapse BPC staging directory. Default: false
   --cbioportal {synapseID}
-                        Optional parameter to specify cbioportal folder
-                        location
-  --production          Whether to run in production mode or not. Default: false
-  --use-grs             Whether to use grs or use dd as primary mapping. Default: false
+                                    Optional parameter to specify cbioportal folder
+                                    location
+  --production                      Whether to run in production mode or not. Default: false
+  --use-grs                         Whether to use grs or use dd as primary mapping. Default: false
+  --cpt_seq_date_replacement_type   The replacement data type to use for the cpt_seq_date value replacement.
+                                     Default: derived_variable
 ```
 
 Example command line:
 
-This runs the release pipeline for BLADDER 1.1 in non-production mode (staging) with GRS enabled.
+This runs the release pipeline for BLADDER 1.1 in non-production mode (staging) with GRS enabled using
+main genie clinical data to do the cpt_seq_date replacement
path: README.md
user: login: rxu17
body: yes the description here mentions `GRS enabled`
created_at: 2025-04-09T23:41:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -579,9 +597,74 @@ def check_oncotree_codes(
             f"There are invalid values in ONCOTREE_CODE column in the clinical df: {invalid_codes}")
 
 
+def replace_cpt_seq_date(
+    input_data : pd.DataFrame, 
+    replacement_data : pd.DataFrame,
+    cpt_seq_date_replacement_type : str
+    ) -> pd.DataFrame:
+    """ Replaces CPT_SEQ_DATE in input BPC file with specified replacement data's 
+        CPT_SEQ_DATE because the values in CPT_SEQ_DATE are incorrect.
+        
+        Current available cpt_seq_date_replacement_type values are ["main_genie", "derived_variable"]
+            - main_genie - uses the main genie consortium release's sample file and 
+            - derived_variable - uses the stats team's provided derived variable file
+                after BPC processing is complete and tables are provided
+
+    Args:
+        input_data (pd.DataFrame): input data with CPT_SEQ_DATE values to be replaced
+        replacement_data (pd.DataFrame): the derived variable data with CPT_SEQ_DATE values to 
+            use as replacement
+        cpt_seq_date_replacement_type (str): the type of replacement used, (e.g: main genie, derived variable)
+    
+    Raises:
+        ValueError: thrown when cpt_seq_date_replacement_type is not one of the valid
+            allowed values
+    
+    Returns:
+        pd.DataFrame: input data with replaced CPT_SEQ_DATE values
+    """
+    # Remove CPT_SEQ_DATE because the values are incorrect
+    del input_data["CPT_SEQ_DATE"]
+    logging.info(f"Replacing CPT_SEQ_DATE values with {cpt_seq_date_replacement_type} values ...")
+    if cpt_seq_date_replacement_type == "main_genie":
+        
+        # main genie clinical is just sample data
+        replacement_data = replacement_data[["SAMPLE_ID", "SEQ_YEAR"]]
+        replacement_data.rename(columns={"SEQ_YEAR": "CPT_SEQ_DATE"}, inplace=True)
+        merge_cols = ["SAMPLE_ID"]
+        
+    elif cpt_seq_date_replacement_type == "derived_variable":
+        # there should only be a unique seq_date per patient id and sample id
+        replacement_data = replacement_data[
+            ["cpt_genie_sample_id", "record_id", "cpt_seq_date"]
path: geniesp/bpc_redcap_export_mapping.py
user: login: rxu17
body: I think that makes sense since `cpt_seq_date` is cancer panel test specific variable thus `sample` specific variable so there should only be a unique `cpt_seq_date` per `cpt_genie_sample_id`
created_at: 2025-04-09T23:42:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -24,27 +25,31 @@ process cBioPortalExport {
          --upload \
          --cbioportal /usr/src/cbioportal \
          --production \
-         --use-grs
+         --use-grs \
+         --cpt_seq_date_replacement_type $cpt_seq_date_replacement_type
path: main.nf
user: login: thomasyu888
body: I didn't look super closely is seq_date the only field we replace from main genie?
created_at: 2025-04-09T23:57:25Z
author_association: MEMBER
diff_hunk: @@ -164,3 +165,259 @@ def test_that_check_oncotree_codes_gives_no_warning_when_all_codes_valid(caplog)
         "There are invalid values in ONCOTREE_CODE column in the clinical df"
         not in caplog.text
     )
+
+
+def test_that_get_derived_variable_file_gets_file_correctly(mock_syn):
+    test_df = pd.DataFrame(
+        dict(
+            cohort = ["BLADDER", "BrCa", "BLADDER"],
+            record_id = ["GENIE-SAGE-1", "GENIE-SAGE-2", "GENIE-SAGE-3"]
+            )
+    )
+    with mock.patch.object(mock_syn, "get") as mock_syn_get, mock.patch.object(
+        pd, "read_csv", return_value = test_df
+        ) as mock_read_csv:
+            output = bpc_export.get_derived_variable_file(
+                mock_syn, 
+                derived_var_synid = "synZZZZ", 
+                cohort = "BLADDER"
+                )
+            assert_frame_equal(
+                output.reset_index(drop=True), pd.DataFrame(
+                    dict(
+                        cohort = ["BLADDER", "BLADDER"],
+                        record_id = ["GENIE-SAGE-1", "GENIE-SAGE-3"]
+                        )
+                ).reset_index(drop=True),
+                check_index_type=False
+            )
+    
+
+@pytest.mark.parametrize(
+    "input, clinical, expected",
+    [
+        (pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "record_id": ["GENIE-1", "GENIE-1"],
+                    "cpt_genie_sample_id": ["GENIE-1-3", "GENIE-1-4"],
+                    "cpt_seq_date": ["2017", "2018"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": [None, None],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "record_id": ["GENIE-1", "GENIE-1"],
+                    "cpt_genie_sample_id": ["GENIE-1-1", "GENIE-1-3"],
+                    "cpt_seq_date": ["2017", "2018"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2017", None],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "record_id": ["GENIE-1", "GENIE-1"],
+                    "cpt_genie_sample_id": ["GENIE-1-1", "GENIE-1-2"],
+                    "cpt_seq_date": ["2017", "2018"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2017", "2018"],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "record_id": ["GENIE-1", "GENIE-1"],
+                    "cpt_genie_sample_id": ["GENIE-1-1", "GENIE-1-2"],
+                    "cpt_seq_date": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2012", "2013"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "record_id": ["GENIE-1", "GENIE-1", "GENIE-1"],
+                    "cpt_genie_sample_id": ["GENIE-1-1", "GENIE-1-2", "GENIE-1-2"],
+                    "cpt_seq_date": ["2014", "2015", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ))
+        ],
+    ids = [
+        "none_replaced", 
+        "some_replaced", 
+        "all_replaced", 
+        "the_same", 
+        "replacement_has_dups"
+        ]
+)
+def test_that_replace_cpt_seq_date_replaces_correctly_with_derived_variable_replacement_type(input, clinical, expected):
+    output = bpc_export.replace_cpt_seq_date(
+        input_data = input, 
+        replacement_data= clinical,
+        cpt_seq_date_replacement_type = "derived_variable"
+        )
+    assert_frame_equal(output, expected)
+
+
+@pytest.mark.parametrize(
+    "input, clinical, expected",
+    [
+        (pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-3", "GENIE-1-4"],
+                    "SEQ_YEAR": ["2017", "2018"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": [None, None],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-3"],
+                    "SEQ_YEAR": ["2017", "2018"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2017", None],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "SEQ_YEAR": ["2017", "2018"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2017", "2018"],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "SEQ_YEAR": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ))
+        ],
+    ids = ["none_replaced", "some_replaced", "all_replaced", "the_same"]
+)
+def test_that_replace_cpt_seq_date_replaces_correctly_with_main_genie_replacement_type(input, clinical, expected):
+    output = bpc_export.replace_cpt_seq_date(
+        input_data = input, 
+        replacement_data= clinical, 
+        cpt_seq_date_replacement_type = "main_genie"
+        )
+    assert_frame_equal(output, expected)
+    
+    
+def test_that_replace_cpt_seq_date_raises_value_error():
+    with pytest.raises(
path: tests/test_bpc_redcap_export_mapping.py
user: login: thomasyu888
body: What's invalid about this?
created_at: 2025-04-09T23:58:28Z
author_association: MEMBER
diff_hunk: @@ -54,26 +54,30 @@ positional arguments:
   release               Specify bpc release (e.g. 1.1-consortium)
 
 optional arguments:
-  -h, --help            Show this help message and exit
-  --upload              Upload files into Synapse BPC staging directory. Default: false
+  -h, --help                        Show this help message and exit
+  --upload                          Upload files into Synapse BPC staging directory. Default: false
   --cbioportal {synapseID}
-                        Optional parameter to specify cbioportal folder
-                        location
-  --production          Whether to run in production mode or not. Default: false
-  --use-grs             Whether to use grs or use dd as primary mapping. Default: false
+                                    Optional parameter to specify cbioportal folder
+                                    location
+  --production                      Whether to run in production mode or not. Default: false
+  --use-grs                         Whether to use grs or use dd as primary mapping. Default: false
+  --cpt_seq_date_replacement_type   The replacement data type to use for the cpt_seq_date value replacement.
+                                     Default: derived_variable
 ```
 
 Example command line:
 
-This runs the release pipeline for BLADDER 1.1 in non-production mode (staging) with GRS enabled.
+This runs the release pipeline for BLADDER 1.1 in non-production mode (staging) with GRS enabled using
+main genie clinical data to do the cpt_seq_date replacement
path: README.md
user: login: danlu1
body: I'm blind.
created_at: 2025-04-10T00:02:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -24,27 +25,31 @@ process cBioPortalExport {
          --upload \
          --cbioportal /usr/src/cbioportal \
          --production \
-         --use-grs
+         --use-grs \
+         --cpt_seq_date_replacement_type $cpt_seq_date_replacement_type
path: main.nf
user: login: rxu17
body: When I looked, it was. Otherwise the main genie clinical files are used to subset out samples and patients from the input data.
created_at: 2025-04-10T00:04:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -164,3 +165,259 @@ def test_that_check_oncotree_codes_gives_no_warning_when_all_codes_valid(caplog)
         "There are invalid values in ONCOTREE_CODE column in the clinical df"
         not in caplog.text
     )
+
+
+def test_that_get_derived_variable_file_gets_file_correctly(mock_syn):
+    test_df = pd.DataFrame(
+        dict(
+            cohort = ["BLADDER", "BrCa", "BLADDER"],
+            record_id = ["GENIE-SAGE-1", "GENIE-SAGE-2", "GENIE-SAGE-3"]
+            )
+    )
+    with mock.patch.object(mock_syn, "get") as mock_syn_get, mock.patch.object(
+        pd, "read_csv", return_value = test_df
+        ) as mock_read_csv:
+            output = bpc_export.get_derived_variable_file(
+                mock_syn, 
+                derived_var_synid = "synZZZZ", 
+                cohort = "BLADDER"
+                )
+            assert_frame_equal(
+                output.reset_index(drop=True), pd.DataFrame(
+                    dict(
+                        cohort = ["BLADDER", "BLADDER"],
+                        record_id = ["GENIE-SAGE-1", "GENIE-SAGE-3"]
+                        )
+                ).reset_index(drop=True),
+                check_index_type=False
+            )
+    
+
+@pytest.mark.parametrize(
+    "input, clinical, expected",
+    [
+        (pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "record_id": ["GENIE-1", "GENIE-1"],
+                    "cpt_genie_sample_id": ["GENIE-1-3", "GENIE-1-4"],
+                    "cpt_seq_date": ["2017", "2018"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": [None, None],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "record_id": ["GENIE-1", "GENIE-1"],
+                    "cpt_genie_sample_id": ["GENIE-1-1", "GENIE-1-3"],
+                    "cpt_seq_date": ["2017", "2018"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2017", None],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "record_id": ["GENIE-1", "GENIE-1"],
+                    "cpt_genie_sample_id": ["GENIE-1-1", "GENIE-1-2"],
+                    "cpt_seq_date": ["2017", "2018"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2017", "2018"],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "record_id": ["GENIE-1", "GENIE-1"],
+                    "cpt_genie_sample_id": ["GENIE-1-1", "GENIE-1-2"],
+                    "cpt_seq_date": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2012", "2013"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "record_id": ["GENIE-1", "GENIE-1", "GENIE-1"],
+                    "cpt_genie_sample_id": ["GENIE-1-1", "GENIE-1-2", "GENIE-1-2"],
+                    "cpt_seq_date": ["2014", "2015", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "PATIENT_ID": ["GENIE-1", "GENIE-1"],
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ))
+        ],
+    ids = [
+        "none_replaced", 
+        "some_replaced", 
+        "all_replaced", 
+        "the_same", 
+        "replacement_has_dups"
+        ]
+)
+def test_that_replace_cpt_seq_date_replaces_correctly_with_derived_variable_replacement_type(input, clinical, expected):
+    output = bpc_export.replace_cpt_seq_date(
+        input_data = input, 
+        replacement_data= clinical,
+        cpt_seq_date_replacement_type = "derived_variable"
+        )
+    assert_frame_equal(output, expected)
+
+
+@pytest.mark.parametrize(
+    "input, clinical, expected",
+    [
+        (pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-3", "GENIE-1-4"],
+                    "SEQ_YEAR": ["2017", "2018"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": [None, None],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-3"],
+                    "SEQ_YEAR": ["2017", "2018"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2017", None],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "SEQ_YEAR": ["2017", "2018"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2017", "2018"],
+                }
+            )),
+        (pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "SEQ_YEAR": ["2014", "2015"],
+                }
+            ),
+         pd.DataFrame(
+                {
+                    "SAMPLE_ID": ["GENIE-1-1", "GENIE-1-2"],
+                    "CPT_SEQ_DATE": ["2014", "2015"],
+                }
+            ))
+        ],
+    ids = ["none_replaced", "some_replaced", "all_replaced", "the_same"]
+)
+def test_that_replace_cpt_seq_date_replaces_correctly_with_main_genie_replacement_type(input, clinical, expected):
+    output = bpc_export.replace_cpt_seq_date(
+        input_data = input, 
+        replacement_data= clinical, 
+        cpt_seq_date_replacement_type = "main_genie"
+        )
+    assert_frame_equal(output, expected)
+    
+    
+def test_that_replace_cpt_seq_date_raises_value_error():
+    with pytest.raises(
path: tests/test_bpc_redcap_export_mapping.py
user: login: rxu17
body: The `cpt_seq_date_replacement_type` here is not one of the available ones
created_at: 2025-04-10T00:05:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -579,9 +597,74 @@ def check_oncotree_codes(
             f"There are invalid values in ONCOTREE_CODE column in the clinical df: {invalid_codes}")
 
 
+def replace_cpt_seq_date(
+    input_data : pd.DataFrame, 
+    replacement_data : pd.DataFrame,
+    cpt_seq_date_replacement_type : str
+    ) -> pd.DataFrame:
+    """ Replaces CPT_SEQ_DATE in input BPC file with specified replacement data's 
+        CPT_SEQ_DATE because the values in CPT_SEQ_DATE are incorrect.
+        
+        Current available cpt_seq_date_replacement_type values are ["main_genie", "derived_variable"]
+            - main_genie - uses the main genie consortium release's sample file and 
+            - derived_variable - uses the stats team's provided derived variable file
+                after BPC processing is complete and tables are provided
+
+    Args:
+        input_data (pd.DataFrame): input data with CPT_SEQ_DATE values to be replaced
+        replacement_data (pd.DataFrame): the derived variable data with CPT_SEQ_DATE values to 
+            use as replacement
+        cpt_seq_date_replacement_type (str): the type of replacement used, (e.g: main genie, derived variable)
+    
+    Raises:
+        ValueError: thrown when cpt_seq_date_replacement_type is not one of the valid
+            allowed values
+    
+    Returns:
+        pd.DataFrame: input data with replaced CPT_SEQ_DATE values
+    """
+    # Remove CPT_SEQ_DATE because the values are incorrect
+    del input_data["CPT_SEQ_DATE"]
+    logging.info(f"Replacing CPT_SEQ_DATE values with {cpt_seq_date_replacement_type} values ...")
+    if cpt_seq_date_replacement_type == "main_genie":
+        
+        # main genie clinical is just sample data
+        replacement_data = replacement_data[["SAMPLE_ID", "SEQ_YEAR"]]
+        replacement_data.rename(columns={"SEQ_YEAR": "CPT_SEQ_DATE"}, inplace=True)
+        merge_cols = ["SAMPLE_ID"]
+        
+    elif cpt_seq_date_replacement_type == "derived_variable":
+        # there should only be a unique seq_date per patient id and sample id
+        replacement_data = replacement_data[
+            ["cpt_genie_sample_id", "record_id", "cpt_seq_date"]
path: geniesp/bpc_redcap_export_mapping.py
user: login: rxu17
body: Done
created_at: 2025-04-10T00:12:07Z
author_association: CONTRIBUTOR

PR comments:
user: login: sonarqubecloud[bot]
created_at: 2025-04-10T00:10:50Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=Sage-Bionetworks_GENIE-Sponsored-Projects&pullRequest=151) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [68 New issues](https://sonarcloud.io/project/issues?id=Sage-Bionetworks_GENIE-Sponsored-Projects&pullRequest=151&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=Sage-Bionetworks_GENIE-Sponsored-Projects&pullRequest=151&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=Sage-Bionetworks_GENIE-Sponsored-Projects&pullRequest=151&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=Sage-Bionetworks_GENIE-Sponsored-Projects&pullRequest=151&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=Sage-Bionetworks_GENIE-Sponsored-Projects&pullRequest=151&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=Sage-Bionetworks_GENIE-Sponsored-Projects&pullRequest=151)


author_association: NONE