Repository: mmcdermott/MEDS_transforms (language: python)

Issue:
Metadata extraction does not appear to be extracting some columns correctly
In particular, `description` and `parent_codes` seem ok, but columns that are part of the `code` constructor in the MIMIC example are not getting pulled properly. For example, in MIMIC, the metadata for the lab itemid `50801` which is used in a code with a `valueuom` of `mm Hg` is recorded in the `d_labitems_to_loinc.csv` as having a `valueuom` of `mm Hg` but in the output extracted metadata it has a null `valueuom`.

Testing should also be expanded to catch this.

Tagging @Jwoo5 for tracking.
Metadata extraction does not appear to be extracting some columns correctly
In particular, `description` and `parent_codes` seem ok, but columns that are part of the `code` constructor in the MIMIC example are not getting pulled properly. For example, in MIMIC, the metadata for the lab itemid `50801` which is used in a code with a `valueuom` of `mm Hg` is recorded in the `d_labitems_to_loinc.csv` as having a `valueuom` of `mm Hg` but in the output extracted metadata it has a null `valueuom`.

Testing should also be expanded to catch this.

Tagging @Jwoo5 for tracking.


Discussion / hints:
I believe with some confidence that the issue that In the extraction function, there is a variable `final_cols` used for the selection of what to include in the output, and it is updated with all of the columns explicitly included for extraction in the metadata config. However, it is not included for columns that are part of the code and not otherwise included in the metadata config, per the code here: https://github.com/mmcdermott/MEDS_transforms/blob/main/src/MEDS_transforms/extract/extract_code_metadata.py#L171 

This means that if you have a metadata config that doesn't include `valueuom`, but `valueuom` is part of the code, such as for some labs (e.g., https://github.com/mmcdermott/MEDS_transforms/blob/main/MIMIC-IV_Example/configs/event_configs.yaml#L86) and then later you have a different metadata block for something else that does include `valueuom` specifically (E.g., https://github.com/mmcdermott/MEDS_transforms/blob/main/MIMIC-IV_Example/configs/event_configs.yaml#L317) then it will end up with a bunch of unexpected nulls in the `valueuom` column because it isn't actually being extracted for most of the events and when they are finally joined the small minority that have it extracted force the column to be included and all others get set to null as though they had included it and it were null.
ok, this doesn't explain the full picture because even those that do have it specified also have nulled valueuom outputs.
I believe with some confidence that the issue that In the extraction function, there is a variable `final_cols` used for the selection of what to include in the output, and it is updated with all of the columns explicitly included for extraction in the metadata config. However, it is not included for columns that are part of the code and not otherwise included in the metadata config, per the code here: https://github.com/mmcdermott/MEDS_transforms/blob/main/src/MEDS_transforms/extract/extract_code_metadata.py#L171 

This means that if you have a metadata config that doesn't include `valueuom`, but `valueuom` is part of the code, such as for some labs (e.g., https://github.com/mmcdermott/MEDS_transforms/blob/main/MIMIC-IV_Example/configs/event_configs.yaml#L86) and then later you have a different metadata block for something else that does include `valueuom` specifically (E.g., https://github.com/mmcdermott/MEDS_transforms/blob/main/MIMIC-IV_Example/configs/event_configs.yaml#L317) then it will end up with a bunch of unexpected nulls in the `valueuom` column because it isn't actually being extracted for most of the events and when they are finally joined the small minority that have it extracted force the column to be included and all others get set to null as though they had included it and it were null.
ok, this doesn't explain the full picture because even those that do have it specified also have nulled valueuom outputs.

Fix patch:
diff --git a/MIMIC-IV_Example/configs/event_configs.yaml b/MIMIC-IV_Example/configs/event_configs.yaml
index 619d5a2e..2986a958 100644
--- a/MIMIC-IV_Example/configs/event_configs.yaml
+++ b/MIMIC-IV_Example/configs/event_configs.yaml
@@ -100,6 +100,7 @@ hosp/labevents:
         description: ["omop_concept_name", "label"] # List of strings are columns to be collated
         itemid: "itemid (omop_source_code)"
         parent_codes: "{omop_vocabulary_id}/{omop_concept_code}"
+        valueuom: "valueuom"
 
 hosp/omr:
   omr:
@@ -218,6 +219,7 @@ icu/chartevents:
         description: ["omop_concept_name", "label"] # List of strings are columns to be collated
         itemid: "itemid (omop_source_code)"
         parent_codes: "{omop_vocabulary_id}/{omop_concept_code}"
+        valueuom: "valueuom"
 
 icu/procedureevents:
   start:


Test patch:
diff --git a/tests/MEDS_Extract/test_extract_code_metadata.py b/tests/MEDS_Extract/test_extract_code_metadata.py
index 74a3d02e..e93bc346 100644
--- a/tests/MEDS_Extract/test_extract_code_metadata.py
+++ b/tests/MEDS_Extract/test_extract_code_metadata.py
@@ -18,30 +18,30 @@
       239684,,HEIGHT,175.271115221764
       239684,"12/28/1980, 00:00:00",DOB,
       239684,"05/11/2010, 17:41:51",ADMISSION//CARDIAC,
-      239684,"05/11/2010, 17:41:51",HR,102.6
+      239684,"05/11/2010, 17:41:51",HR//bpm,102.6
       239684,"05/11/2010, 17:41:51",TEMP,96.0
-      239684,"05/11/2010, 17:48:48",HR,105.1
+      239684,"05/11/2010, 17:48:48",HR//bpm,105.1
       239684,"05/11/2010, 17:48:48",TEMP,96.2
-      239684,"05/11/2010, 18:25:35",HR,113.4
+      239684,"05/11/2010, 18:25:35",HR//bpm,113.4
       239684,"05/11/2010, 18:25:35",TEMP,95.8
-      239684,"05/11/2010, 18:57:18",HR,112.6
+      239684,"05/11/2010, 18:57:18",HR//bpm,112.6
       239684,"05/11/2010, 18:57:18",TEMP,95.5
       239684,"05/11/2010, 19:27:19",DISCHARGE,
       1195293,,EYE_COLOR//BLUE,
       1195293,,HEIGHT,164.6868838269085
       1195293,"06/20/1978, 00:00:00",DOB,
       1195293,"06/20/2010, 19:23:52",ADMISSION//CARDIAC,
-      1195293,"06/20/2010, 19:23:52",HR,109.0
+      1195293,"06/20/2010, 19:23:52",HR//bpm,109.0
       1195293,"06/20/2010, 19:23:52",TEMP,100.0
-      1195293,"06/20/2010, 19:25:32",HR,114.1
+      1195293,"06/20/2010, 19:25:32",HR//bpm,114.1
       1195293,"06/20/2010, 19:25:32",TEMP,100.0
-      1195293,"06/20/2010, 19:45:19",HR,119.8
+      1195293,"06/20/2010, 19:45:19",HR//bpm,119.8
       1195293,"06/20/2010, 19:45:19",TEMP,99.9
-      1195293,"06/20/2010, 20:12:31",HR,112.5
+      1195293,"06/20/2010, 20:12:31",HR//bpm,112.5
       1195293,"06/20/2010, 20:12:31",TEMP,99.8
-      1195293,"06/20/2010, 20:24:44",HR,107.7
+      1195293,"06/20/2010, 20:24:44",HR//bpm,107.7
       1195293,"06/20/2010, 20:24:44",TEMP,100.0
-      1195293,"06/20/2010, 20:41:33",HR,107.5
+      1195293,"06/20/2010, 20:41:33",HR//bpm,107.5
       1195293,"06/20/2010, 20:41:33",TEMP,100.4
       1195293,"06/20/2010, 20:50:04",DISCHARGE,
 
@@ -52,14 +52,14 @@
       68729,,HEIGHT,160.3953106166676
       68729,"03/09/1978, 00:00:00",DOB,
       68729,"05/26/2010, 02:30:56",ADMISSION//PULMONARY,
-      68729,"05/26/2010, 02:30:56",HR,86.0
+      68729,"05/26/2010, 02:30:56",HR//bpm,86.0
       68729,"05/26/2010, 02:30:56",TEMP,97.8
       68729,"05/26/2010, 04:51:52",DISCHARGE,
       814703,,EYE_COLOR//HAZEL,
       814703,,HEIGHT,156.48559093209357
       814703,"03/28/1976, 00:00:00",DOB,
       814703,"02/05/2010, 05:55:39",ADMISSION//ORTHOPEDIC,
-      814703,"02/05/2010, 05:55:39",HR,170.2
+      814703,"02/05/2010, 05:55:39",HR//bpm,170.2
       814703,"02/05/2010, 05:55:39",TEMP,100.1
       814703,"02/05/2010, 07:02:30",DISCHARGE,
 
@@ -69,7 +69,7 @@
       754281,,HEIGHT,166.22261567137025
       754281,"12/19/1988, 00:00:00",DOB,
       754281,"01/03/2010, 06:27:59",ADMISSION//PULMONARY,
-      754281,"01/03/2010, 06:27:59",HR,142.0
+      754281,"01/03/2010, 06:27:59",HR//bpm,142.0
       754281,"01/03/2010, 06:27:59",TEMP,99.8
       754281,"01/03/2010, 08:22:13",DISCHARGE,
 
@@ -79,11 +79,11 @@
       1500733,,HEIGHT,158.60131573580904
       1500733,"07/20/1986, 00:00:00",DOB,
       1500733,"06/03/2010, 14:54:38",ADMISSION//ORTHOPEDIC,
-      1500733,"06/03/2010, 14:54:38",HR,91.4
+      1500733,"06/03/2010, 14:54:38",HR//bpm,91.4
       1500733,"06/03/2010, 14:54:38",TEMP,100.0
-      1500733,"06/03/2010, 15:39:49",HR,84.4
+      1500733,"06/03/2010, 15:39:49",HR//bpm,84.4
       1500733,"06/03/2010, 15:39:49",TEMP,100.3
-      1500733,"06/03/2010, 16:20:49",HR,90.1
+      1500733,"06/03/2010, 16:20:49",HR//bpm,90.1
       1500733,"06/03/2010, 16:20:49",TEMP,100.1
       1500733,"06/03/2010, 16:44:26",DISCHARGE,
     """
@@ -91,9 +91,9 @@
 
 
 INPUT_METADATA_FILE = """
-lab_code,title,loinc
-HR,Heart Rate,8867-4
-temp,Body Temperature,8310-5
+lab_code,valueuom,title,loinc
+HR,bpm,Heart Rate,8867-4
+temp,,Body Temperature,8310-5
 """
 
 DEMO_METADATA_FILE = """
@@ -135,7 +135,9 @@
     time: col(disch_date)
     time_format: "%m/%d/%Y, %H:%M:%S"
   HR:
-    code: HR
+    code:
+      - HR
+      - col(valueuom)
     time: col(vitals_date)
     time_format: "%m/%d/%Y, %H:%M:%S"
     numeric_value: HR
@@ -143,6 +145,7 @@
       input_metadata:
         description: {"title": {"lab_code": "HR"}}
         parent_codes: {"LOINC/{loinc}": {"lab_code": "HR"}}
+        valueuom: {"valueuom": {"lab_code": "HR"}} # If we didn't have this valueuom would be null for HR rows
   temp:
     code: TEMP
     time: col(vitals_date)
@@ -152,6 +155,7 @@
       input_metadata:
         description: {"title": {"lab_code": "temp"}}
         parent_codes: {"LOINC/{loinc}": {"lab_code": "temp"}}
+        valueuom: {"valueuom": {"lab_code": "temp"}}
 """
 
 SHARDS_JSON = {
@@ -164,7 +168,7 @@
 WANT_OUTPUTS = {
     "metadata/codes": pl.DataFrame(
         {
-            "code": ["EYE_COLOR//BLUE", "EYE_COLOR//BROWN", "EYE_COLOR//HAZEL", "HR", "TEMP"],
+            "code": ["EYE_COLOR//BLUE", "EYE_COLOR//BROWN", "EYE_COLOR//HAZEL", "HR//bpm", "TEMP"],
             "description": [
                 "Blue Eyes. Less common than brown.",
                 "Brown Eyes. The most common eye color.",
@@ -173,6 +177,7 @@
                 "Body Temperature",
             ],
             "parent_codes": [None, None, None, ["LOINC/8867-4"], ["LOINC/8310-5"]],
+            "valueuom": [None, None, None, "bpm", None],
         }
     ),
 }