Repository: pyjanitor-devs/pyjanitor

Issue:
convert_excel_date throws error when input contains strings
After importing an excel sheet into a dataframe, applying the convert_excel_date throws an error, `ValueError: unit must not be specified if the input contains a str`, if the columns are objects with strings. The error comes from the [pandas.to_timedelta](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_timedelta.html) function, which is rather confusing for the user at first.

Converting the column to numeric and then dropping inf and nan values fixes this issue (e.g. `df.column..apply(pd.to_numeric, errors='coerce',downcast='integer').astype('float').astype('Int64')` and then `df = df[~df[column].isin([np.nan, np.inf, -np.inf]).any(1)]`).

Perhaps the function should trap for this error and either convert when appropriate, remove the unit input when appropriate, or at the least tell the user what to do to fix the issue.

Also of note, the above conversion does a float and then an Int64 conversion because the Int64 conversion alone will fail if infinite or null values are in the set. The nan/inf drop is then necessary because the to_timedelta seems to require `int64` not `Int64` (the latter allowing nan values). Suggesting to the user that they should run the above first, and noting the null/inf issue, may be the best way to address this issue and trap for the relevant errors.

<!-- Please provide a brief description of your bug. Do NOT paste the stack trace here. -->

# System Information

<!-- System information helps us. To keep things simple, just let us know the OS and Python version first.
You can provide the optional information later. -->

- Operating system: Windows  <!-- delete the appropriate ones -->
- Python version (required): 3.8

# Minimally Reproducible Code

`df_train=df_train.convert_excel_date('column_name')`

Unfortunately I can't share my excel sheet in this case to reproduce the error (proprietary), but importing date columns as objects that are string values and then running the above function on that column causes the error.


Discussion:
@tomshaffner thanks for the feedback. Do you mind submitting a PR for this?
Thanks for reporting this issue, @tomshaffner! I have to admit that I am not too familiar with the `convert_excel_date` function, as I rarely do much in the way of timeseries problems. That said, your issue description is quite clear. Would you be kind enough to help me with one little thing - could you edit your post with a minimal copy/paste-able CSV file that guarantees an error is thrown? That will help in a few areas:

1. Developing the bug fix.
2. Adding a software test so we never have to deal with the same error again.

Next week, when I'm officially on paternity leave, I might be able to find some downtime in between diaper changes to take a stab at this problem. Alternatively, if you'd like to try your hand at a PR, I would suggest the following:

1. The desired behaviour with data is that invalid data types where a particular one is expected should error loudly. Since `convert_excel_date` expects an integer, then one thing we could do is simply check that the column of interest is of integer type and fail loudly stating that there are non-integer elements.
2. To suggest an action, we could recommend that a user install `pandera` and use it to validate their data. `pandera` is awesome because it'll tell you which exact rows + columns fail data validation checks. (The act of data validation, which is distinct from cleaning, is something we don't implement in the pyjanitor library because pandera does it so well.)
3. Coercion could be a flag, in which case we'd do as your code suggests. I'd have to check that the complexity of the code, and hence the maintainability, doesn't increase by adding a flag. (That relates to feature creep.) I'm generally a follower of the Zen of Python, where simplicity is better than complexity, so unless the implementation is clean and slick, I'd probably not allow this into the codebase. 

The above are notes for myself, but if you wanted to try your hand at a PR, I'd love to review it too! Otherwise, hang tight :smile:.

Resolving patch:
diff --git a/janitor/functions.py b/janitor/functions.py
--- a/janitor/functions.py
+++ b/janitor/functions.py
@@ -24,6 +24,7 @@
 
 import numpy as np
 import pandas as pd
+from pandas.api.types import is_numeric_dtype
 import pandas_flavor as pf
 from multipledispatch import dispatch
 from natsort import index_natsorted
@@ -1089,7 +1090,15 @@ def convert_excel_date(
     :param df: A pandas DataFrame.
     :param column_name: A column name.
     :returns: A pandas DataFrame with corrected dates.
+    :raises ValueError: if There are non numeric values in the column.
     """  # noqa: E501
+
+    if not is_numeric_dtype(df[column_name]):
+        raise ValueError(
+            "There are non-numeric values in the column. \
+    All values must be numeric"
+        )
+
     df[column_name] = pd.TimedeltaIndex(
         df[column_name], unit="d"
     ) + dt.datetime(


Test patch:
diff --git a/tests/functions/test_convert_excel_date.py b/tests/functions/test_convert_excel_date.py
--- a/tests/functions/test_convert_excel_date.py
+++ b/tests/functions/test_convert_excel_date.py
@@ -18,3 +18,15 @@ def test_convert_excel_date():
     )
 
     assert df["hire_date"].dtype == "M8[ns]"
+
+
+@pytest.mark.functions
+def test_convert_excel_date_with_string_data():
+    """ Raises ValueError if values of column are not numeric """
+    df = pd.read_excel(
+        Path(pytest.EXAMPLES_DIR) / "notebooks" / "dirty_data.xlsx",
+        engine="openpyxl",
+    ).clean_names()
+
+    with pytest.raises(ValueError):
+        df.convert_excel_date("hire_date_str")