Repository: domoinc/domo-python-sdk
Pull request #111: Add use schema feature
Author: login: samrands
Created: 2025-04-18T18:45:06Z | Merged: 2025-04-21T16:00:30Z

See https://github.com/domoinc/domo-python-sdk/pull/109.


Diff:
diff --git a/pydomo/__init__.py b/pydomo/__init__.py
index 89cf4da..9a73c63 100644
--- a/pydomo/__init__.py
+++ b/pydomo/__init__.py
@@ -13,7 +13,6 @@
 from pydomo.utilities import UtilitiesClient
 from pandas import read_csv
 from pandas import DataFrame
-from pandas import to_datetime
 from io import StringIO
 import logging
 import json
@@ -178,7 +177,7 @@ def ds_query(self, dataset_id, query, return_data=True):
         return output
 
 
-    def ds_get(self, dataset_id):
+    def ds_get(self, dataset_id, use_schema=True) -> DataFrame:
         """
             Export data to pandas Dataframe
 
@@ -186,27 +185,45 @@ def ds_get(self, dataset_id):
             >>> print(df.head())
 
             :Parameters:
-            - `dataset_id`:     id of a dataset (str)
-
+            - `dataset_id`: id of a dataset (str)
+            - `use_schema`: whether to use the dataset schema to determine column types (bool, default True)
+                                if false, let pandas dynamically determine types from the data
             :Returns:
             pandas dataframe
         """
         csv_download = self.datasets.data_export(dataset_id, include_csv_header=True)
-
         content = StringIO(csv_download)
-        df = read_csv(content)
-
-        # Convert to dates or datetimes if possible
-        for col in df.columns:
-            if df[col].dtype == 'object':
-                try:
-                    df[col] = to_datetime(df[col])
-                except ValueError:
-                    pass
-                except TypeError:
-                    pass
-
-        return df
+
+        if use_schema:
+            try:
+                schema_dict = self.ds_meta(dataset_id)
+
+                if "schema" in schema_dict and "columns" in schema_dict["schema"]:
+
+                    dtype_dict = {}
+                    date_columns = []
+
+                    for column in schema_dict["schema"]["columns"]:
+                        col_name = column["name"]
+                        col_type = column["type"]
+
+                        dtype_dict[col_name] = self.utilities.convert_domo_type_to_pandas_type(col_type)
+
+                        if self.utilities.is_date_type(col_type):
+                            date_columns.append(col_name)
+
+                    return read_csv(
+                        content, dtype=dtype_dict, parse_dates=date_columns
+                    )
+
+            except Exception as err:
+                print(f"""An error occurred while converting domo schema to pandas dtypes. 
+                Letting pandas attempt to read the data. Error={err}""")
+
+                content.seek(0)
+
+        return self.utilities.read_content_to_dataframe(content)
+
     
     def ds_get_dict(self,ds_id):
         my_data = self.datasets.data_export(ds_id,True)
diff --git a/pydomo/utilities/UtilitiesClient.py b/pydomo/utilities/UtilitiesClient.py
index e65eda4..d6418dd 100644
--- a/pydomo/utilities/UtilitiesClient.py
+++ b/pydomo/utilities/UtilitiesClient.py
@@ -3,6 +3,8 @@
 import math
 import sys
 import json
+from pandas import read_csv
+from pandas import to_datetime
 
 from pydomo.DomoAPIClient import DomoAPIClient
 from pydomo.datasets import DataSetClient
@@ -37,6 +39,38 @@ def typeConversionText(self, dt):
 
         return result
 
+    def convert_domo_type_to_pandas_type(self, domo_type):
+        # type mapping can be adjusted accordingly
+        # there is no Date type in Pandas and Timestamps are not a type in Domo
+        type_mapping = {
+            "LONG": "Int64",
+            "STRING": "string",
+            "DECIMAL": "Float64",
+            "DOUBLE": "Float64",
+            "DATETIME": "datetime64[ns]",
+            "DATE": "datetime64[ns]",
+        }
+
+        return type_mapping.get(domo_type, "object")
+
+    def is_date_type(self, column_type):
+        return column_type in ("DATE", "DATETIME")
+
+    def read_content_to_dataframe(self, content):
+        df = read_csv(content)
+
+        # Convert to dates or datetimes if possible
+        for col in df.columns:
+            if df[col].dtype == "object":
+                try:
+                    df[col] = to_datetime(df[col])
+                except ValueError:
+                    pass
+                except TypeError:
+                    pass
+
+        return df
+
     def identical(self, c1, c2):
         cc1 = json.dumps(c1)
         cc2 = json.dumps(c2)


Reviews:
user: login: samrands
body: Overall looks great.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-18T18:48:41Z
user: login: erikjamesmason
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-21T16:58:58Z

Review comments:
diff_hunk: @@ -178,35 +177,55 @@ def ds_query(self, dataset_id, query, return_data=True):
         return output
 
 
-    def ds_get(self, dataset_id):
+    def ds_get(self, dataset_id, use_schema=True) -> DataFrame:
         """
             Export data to pandas Dataframe
 
             >>> df = domo.ds_get('80268aef-e6a1-44f6-a84c-f849d9db05fb')
             >>> print(df.head())
 
             :Parameters:
-            - `dataset_id`:     id of a dataset (str)
-
+            - `dataset_id`: id of a dataset (str)
+            - `use_schema`: whether to use the dataset schema to determine column types (bool, default True)
+                                if false, let pandas dynamically determine types from the data
             :Returns:
             pandas dataframe
         """
         csv_download = self.datasets.data_export(dataset_id, include_csv_header=True)
-
         content = StringIO(csv_download)
-        df = read_csv(content)
-
-        # Convert to dates or datetimes if possible
-        for col in df.columns:
-            if df[col].dtype == 'object':
-                try:
-                    df[col] = to_datetime(df[col])
-                except ValueError:
-                    pass
-                except TypeError:
-                    pass
-
-        return df
+
+        if use_schema:
+            try:
+                schema_dict = self.ds_meta(dataset_id)
+
+                if "schema" in schema_dict and "columns" in schema_dict["schema"]:
+
+                    dtype_dict = {}
+                    date_columns = []
+
+                    for column in schema_dict["schema"]["columns"]:
+                        col_name = column["name"]
+                        col_type = column["type"]
+
+                        dtype_dict[col_name] = self.utilities.convert_domo_type_to_pandas_type(col_type)
+
+                        if self.utilities.is_date_type(col_type):
+                            date_columns.append(col_name)
+
+                    return read_csv(
+                        content, dtype=dtype_dict, parse_dates=date_columns
path: pydomo/__init__.py
user: login: samrands
body: I removed this if-block since I don't think it's necessary. I'll admit, I'm not a Python expert so maybe there's something I'm missing. I did run this code for a datasource that didn't have any date columns and it didn't break so I think it's fine.
created_at: 2025-04-18T18:47:47Z
author_association: COLLABORATOR
diff_hunk: @@ -178,35 +177,55 @@ def ds_query(self, dataset_id, query, return_data=True):
         return output
 
 
-    def ds_get(self, dataset_id):
+    def ds_get(self, dataset_id, use_schema=True) -> DataFrame:
         """
             Export data to pandas Dataframe
 
             >>> df = domo.ds_get('80268aef-e6a1-44f6-a84c-f849d9db05fb')
             >>> print(df.head())
 
             :Parameters:
-            - `dataset_id`:     id of a dataset (str)
-
+            - `dataset_id`: id of a dataset (str)
+            - `use_schema`: whether to use the dataset schema to determine column types (bool, default True)
+                                if false, let pandas dynamically determine types from the data
             :Returns:
             pandas dataframe
         """
         csv_download = self.datasets.data_export(dataset_id, include_csv_header=True)
-
         content = StringIO(csv_download)
-        df = read_csv(content)
-
-        # Convert to dates or datetimes if possible
-        for col in df.columns:
-            if df[col].dtype == 'object':
-                try:
-                    df[col] = to_datetime(df[col])
-                except ValueError:
-                    pass
-                except TypeError:
-                    pass
-
-        return df
+
+        if use_schema:
+            try:
+                schema_dict = self.ds_meta(dataset_id)
+
+                if "schema" in schema_dict and "columns" in schema_dict["schema"]:
+
+                    dtype_dict = {}
+                    date_columns = []
+
+                    for column in schema_dict["schema"]["columns"]:
+                        col_name = column["name"]
+                        col_type = column["type"]
+
+                        dtype_dict[col_name] = self.utilities.convert_domo_type_to_pandas_type(col_type)
+
+                        if self.utilities.is_date_type(col_type):
+                            date_columns.append(col_name)
+
+                    return read_csv(
+                        content, dtype=dtype_dict, parse_dates=date_columns
path: pydomo/__init__.py
user: login: erikjamesmason
body: i think you're referencing the date/datetime and date_columns handling - that should be fine, it was throwing an error previously when parse_dates and dtypes were not handled separately and specifically for datetime64[ns] but the removal and usage from the utilities.is_date_type should accomplish the same thing (as you noticed)
created_at: 2025-04-21T16:58:58Z
author_association: NONE

PR comments:
user: login: erikjamesmason
created_at: 2025-04-21T23:18:12Z
body: ah, @samrands i found the error i was getting, hence why i had the separated if-block:

`An error occurred while converting domo schema to pandas dtypes. 
                Letting pandas attempt to read the data. Error=the dtype datetime64[ns] is not supported for parsing, pass this column using parse_dates instead`
                
i'll rework it again to better for your utility helper convention to make sure this error doesn't reoccur
author_association: NONE
user: login: samrands
created_at: 2025-04-22T17:29:52Z
body: @erikjamesmason I'm happy to make the code change, I'm just confused on how it's occurring. How is it possible to have an entry in dtype_dict of datetime64[ns] and not also have that column in date_columns?
author_association: COLLABORATOR
user: login: samrands
created_at: 2025-04-22T17:31:28Z
body: I guess I can see how it would occur on your old branch, since the if-block where you add the column to date_columns is not indented to be within the for loop. So, effectively, only the last column can be included in date_columns. But I'm not sure how that'd occur on my branch.
author_association: COLLABORATOR
user: login: erikjamesmason
created_at: 2025-04-22T17:42:14Z
body: i'll double check tonight but i made a fresh env with pip for newest version of PyDomo and checked the version in pip list all in a new workspace, but who knows - maybe since i was testing in a notebook, maybe the envs got convoluted somehow. i'll set up some tests and see if i can replicate it again, then update here. 
also, i believe actually had another commit that fixed that loop indentation issue but since you had reformed it, i deleted the branch - but my bad for sending over the faulty one
author_association: NONE
user: login: erikjamesmason
created_at: 2025-04-22T17:42:44Z
body: either way, appreciate the time and effort on this, sorry it's become a hassle
author_association: NONE
user: login: samrands
created_at: 2025-04-22T17:45:01Z
body: You're welcome. I mean it's really just caused by me thinking I knew better than I actually did. But I do want to fully understand why before making the change just to be sure.
author_association: COLLABORATOR
user: login: samrands
created_at: 2025-05-23T15:09:56Z
body: Hi @erikjamesmason (for some reason I can't @ you on the other PR), but you were right that there was a bug, and it should be fixed with v0.3.0.13 (https://github.com/domoinc/domo-python-sdk/pull/112).
author_association: COLLABORATOR
user: login: erikjamesmason
created_at: 2025-05-23T15:21:30Z
body: thanks for the update @samrands - i'll be sure to check it out but very much appreciate you taking the time to address it!
author_association: NONE