Repository: innobi/pantab
Pull request #22 (2019-11-15 21:50:34)

Improved Error Message for Bad Object Data


Patch:
diff --git a/doc/source/caveats.rst b/doc/source/caveats.rst
index fff13af..42d1ccd 100644
--- a/doc/source/caveats.rst
+++ b/doc/source/caveats.rst
@@ -1,6 +1,51 @@
 Usage Limitations
 =================
 
+Type Mapping
+------------
+
+pantab maps the following dtypes from pandas to the equivalent column type in Tableau.
+
++--------------------+-----------------+------------+
+|Dtype               |Tableau Type     |Nullability |
++====================+=================+============+
+|int16               |SMALLINT         |NOT NULLABLE|
++--------------------+-----------------+------------+
+|int32               |INT              |NOT NULLABLE|
++--------------------+-----------------+------------+
+|int64               |BIGINT           |NOT NULLABLE|
++--------------------+-----------------+------------+
+|Int16               |SMALLINT         |NULLABLE    |
++--------------------+-----------------+------------+
+|Int32               |INT              |NULLABLE    |
++--------------------+-----------------+------------+
+|Int64               |BIGINT           |NULLABLE    |
++--------------------+-----------------+------------+
+|float32             |DOUBLE           |NULLABLE    |
++--------------------+-----------------+------------+
+|float64             |DOUBLE           |NULLABLE    |
++--------------------+-----------------+------------+
+|bool                |BOOL             |NOT NULLABLE|
++--------------------+-----------------+------------+
+|datetime64[ns]      |TIMESTAMP        |NULLABLE    |
++--------------------+-----------------+------------+
+|datetime64[ns, UTC] |TIMESTAMP_WITH_TZ|Nullable    |
++--------------------+-----------------+------------+
+|timedelta64[ns]     |INTERVAL         |NULLABLE    |
++--------------------+-----------------+------------+
+|object              |TEXT             |NULLABLE    |
++--------------------+-----------------+------------+
+
+Any dtype not explicitly listed in the above table will raise a ValueError if trying to write out data.
+
+.. warning::
+
+   ``object`` dtype is a very generic type in pandas, but is the closest type to a native "text" type available prior to 1.0.0. Passing non-string / NULL data through an ``object`` dtype will raise a TypeError, though it is perfectly valid for storage in pandas. This may change in the future as pandas 1.0.0 introduces a dedicated "String" data type.
+
+.. note::
+
+   All objects can maintain their type when "round-tripping" to/from Hyper extracts, with the exception of the float32 object as only DOUBLE is available for floating point storage in Hyper. 
+
 Missing Value Handling
 ----------------------
 In the scientific Python community missing values are often notated as ``np.nan``, which is a float value implementing the `IEEE 754 <https://en.wikipedia.org/wiki/IEEE_754>`_ standard for missing value handling. Because NumPy stores homogenous arrays, there has been a long standing issue in pandas where integers could not store missing records, as noted in the `pandas documentation on missing values <https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#working-with-missing-data>`_.
@@ -24,8 +69,3 @@ Timedelta Components
 --------------------
 
 The ``pd.Timedelta`` and the ``Interval`` exposed by the Hyper API have similar but different storage mechanisms that may cause inconsistencies. Specifically, a `pd.Timedelta` does not have a month component to it, so reading ``Interval`` objects from a Hyper Extract that have this component will raise a ``TypeError``.  The Hyper API's ``Interval`` only offers storage of days and microseconds (aside from months). pantab will convert hours, minutes, seconds, etc... into microseconds for you, but reading that information back from a Hyper extract is lossy and will only provide back the microsecond storage.
-
-float32 Type Preservation
--------------------------
-
-The Hyper API does not offer SINGLE precision storage. As such, any float32 objects will be upcast to 64 bits when writing to an extract. When reading, no attempt is made to downsize for you, so you will always get float64 back from a Hyper extract.
diff --git a/pantab/_pantab.py b/pantab/_pantab.py
index 5c5c456..f325a75 100644
--- a/pantab/_pantab.py
+++ b/pantab/_pantab.py
@@ -191,13 +191,26 @@ def _insert_frame(
             df.iloc[:, index] = content.apply(_timedelta_to_interval)
 
     with tab_api.Inserter(connection, table_def) as inserter:
-        for row in df.itertuples(index=False):
-            for index, val in enumerate(row):
+        for row_index, row in enumerate(df.itertuples(index=False)):
+            for col_index, val in enumerate(row):
                 # Missing value handling
                 if val is None or val != val:
                     inserter._Inserter__write_null()
                 else:
-                    getattr(inserter, insert_funcs[index])(val)
+                    try:
+                        getattr(inserter, insert_funcs[col_index])(val)
+                    except TypeError as e:
+                        column = df.iloc[:, col_index]
+                        msg = (
+                            f"Unsupported type '{type(val)}' for column type "
+                            f"'{column.dtype}' (column '{column.name}' row {row_index})"
+                        )
+
+                        msg += (
+                            "\n See https://pantab.readthedocs.io/en/latest/"
+                            "caveats.html#type-mapping"
+                        )
+                        raise TypeError(msg) from e
 
         inserter.execute()
 
diff --git a/pantab/newsfragments/19.feature b/pantab/newsfragments/19.feature
new file mode 100644
index 0000000..e5200b5
--- /dev/null
+++ b/pantab/newsfragments/19.feature
@@ -0,0 +1,1 @@
+Improved error messaging when attempting to write invalid data.
\ No newline at end of file


Test patch:
diff --git a/pantab/tests/test_pantab.py b/pantab/tests/test_pantab.py
index d613e02..3f89a35 100644
--- a/pantab/tests/test_pantab.py
+++ b/pantab/tests/test_pantab.py
@@ -267,3 +267,15 @@ def test_months_in_interval_raises(df, tmp_hyper, monkeypatch):
         ValueError, match=r"Cannot read Intervals with month componenets\."
     ):
         pantab.frames_from_hyper(tmp_hyper)
+
+
+def test_bad_value_gives_clear_message(tmp_hyper):
+    df = pd.DataFrame([[{"a": "b"}]], columns=["a"])
+
+    msg = (
+        r"Unsupported type '<class 'dict'>' for column type 'object' "
+        r"\(column 'a' row 0\)"
+    )
+
+    with pytest.raises(TypeError, match=msg):
+        pantab.frame_to_hyper(df, tmp_hyper, table="test")