Repository: DrGFreeman/dynamo-pandas (language: python)

Issue:
Timedelta string values cannot be converted with the dtype parameter
Timedelta string values stored in a table cannot be converted with the `dtype` parameter of the `get_df` and `to_df` functions or using the dataframe `astype` method.  This is due to a known bug in pandas (ref.: https://github.com/pandas-dev/pandas/issues/38509).

As a result, unit tests for the `dtype` parameter of the `get_df` and `put_df` function do not test this conversion. Once the pandas issue is resolved, this conversion can be added to the tests.

As a workaround, the Timedelta columns can be converted using `pd.to_timedelta(df.column_name)`.


Discussion / hints:
Tests to update:

https://github.com/DrGFreeman/dynamo-pandas/blob/72df84ba2847296bd7207b5cd55832581885ea38/tests/test_main.py#L122

https://github.com/DrGFreeman/dynamo-pandas/blob/72df84ba2847296bd7207b5cd55832581885ea38/tests/test_main.py#L221
Docs to update:

- [README.md](https://github.com/DrGFreeman/dynamo-pandas/blob/main/README.md)
- [docs/overview.rst](https://github.com/DrGFreeman/dynamo-pandas/blob/main/docs/overview.rst)
- https://github.com/DrGFreeman/dynamo-pandas/blob/99e09c71b7bafeb21dc11a0785cabeec3693ba79/dynamo_pandas/dynamo_pandas.py#L83-L101


Fix patch:
diff --git a/README.md b/README.md
index 64ba7ce..258191d 100644
--- a/README.md
+++ b/README.md
@@ -16,8 +16,8 @@ The project's documentation is available at https://dynamo-pandas.readthedocs.io
 
 
 ## Requirements
-* `python>=3.7`
-* `pandas>=1`
+* `python>=3.8`
+* `pandas>=1.2`
 * `boto3`
 
 ## Installation
@@ -59,7 +59,7 @@ The columns of the dataframe use different data types, some of which are not nat
 RangeIndex: 4 entries, 0 to 3
 Data columns (total 5 columns):
     #   Column        Non-Null Count  Dtype          
----  ------        --------------  -----          
+   ---  ------        --------------  -----          
     0   player_id     4 non-null      object         
     1   last_play     4 non-null      datetime64[ns] 
     2   play_time     4 non-null      timedelta64[ns]
@@ -134,12 +134,26 @@ The `dtype` parameter of the `get_df` function allows specifying the desired dat
 ...     dtype={
 ...         "bonus_points": "Int8",
 ...         "last_play": "datetime64[ns, UTC]",
-...         # "play_time": "timedelta64[ns]"  # See note below.
+...         "play_time": "timedelta64[ns]"  # See note below.
 ...     }
 ... )
+>>> df.info()
+
+<class 'pandas.core.frame.DataFrame'>
+RangeIndex: 2 entries, 0 to 1
+Data columns (total 5 columns):
+    #   Column        Non-Null Count  Dtype              
+   ---  ------        --------------  -----              
+    0   bonus_points  1 non-null      Int8               
+    1   player_id     2 non-null      object             
+    2   last_play     2 non-null      datetime64[ns, UTC]
+    3   rating        2 non-null      float64            
+    4   play_time     2 non-null      timedelta64[ns]    
+dtypes: Int8(1), datetime64[ns, UTC](1), float64(1), object(1), timedelta64[ns](1)
+memory usage: 196.0+ bytes
 ```
 
-**Note**: Due to a known bug in pandas, timedelta strings cannot currently be converted back to Timedelta type via this parameter (ref. https://github.com/pandas-dev/pandas/issues/38509). Use the pandas.to_timedelta function instead:
+**Note**: Due to a known bug in pandas versions < 1.5, timedelta strings cannot be converted back to Timedelta type via this parameter (ref. https://github.com/pandas-dev/pandas/issues/38509). If using pandas < 1.5, use the pandas.to_timedelta function instead:
 
 
 ```python
@@ -150,7 +164,7 @@ The `dtype` parameter of the `get_df` function allows specifying the desired dat
 RangeIndex: 2 entries, 0 to 1
 Data columns (total 5 columns):
     #   Column        Non-Null Count  Dtype              
----  ------        --------------  -----              
+   ---  ------        --------------  -----              
     0   bonus_points  1 non-null      Int8               
     1   player_id     2 non-null      object             
     2   last_play     2 non-null      datetime64[ns, UTC]
diff --git a/docs/overview.rst b/docs/overview.rst
index 784e274..ce70545 100644
--- a/docs/overview.rst
+++ b/docs/overview.rst
@@ -31,7 +31,7 @@ The columns of the dataframe use different data types, some of which are not nat
 RangeIndex: 4 entries, 0 to 3
 Data columns (total 5 columns):
     #   Column        Non-Null Count  Dtype          
----  ------        --------------  -----          
+   ---  ------        --------------  -----          
     0   player_id     4 non-null      object         
     1   last_play     4 non-null      datetime64[ns] 
     2   play_time     4 non-null      timedelta64[ns]
@@ -91,7 +91,7 @@ The ``dtype`` parameter of the ``get_df`` function allows specifying the desired
 ...     dtype={
 ...         "bonus_points": "Int8",
 ...         "last_play": "datetime64[ns, UTC]",
-...         # "play_time": "timedelta64[ns]"  # See note below.
+...         "play_time": "timedelta64[ns]"  # See note below.
 ...     }
 ... )
 >>> df.info()
@@ -104,11 +104,11 @@ Data columns (total 5 columns):
     1   player_id     2 non-null      object             
     2   last_play     2 non-null      datetime64[ns, UTC]
     3   rating        2 non-null      float64            
-    4   play_time     2 non-null      object
-dtypes: Int8(1), datetime64[ns, UTC](1), float64(1), object(2)
+    4   play_time     2 non-null      timedelta64[ns]
+dtypes: Int8(1), datetime64[ns, UTC](1), float64(1), object(1), timedelta64[ns](1)
 memory usage: 196.0+ bytes
 
-.. note:: Due to a `known bug in pandas <https://github.com/pandas-dev/pandas/issues/38509>`_, timedelta strings cannot currently be converted back to timedelta64 type via the ``dtype`` parameter. Use the ``pandas.to_timedelta`` function instead:
+.. note:: Due to a `known bug in pandas versions < 1.5 <https://github.com/pandas-dev/pandas/issues/38509>`_, timedelta strings cannot be converted back to timedelta64 type via the ``dtype`` parameter. If using pandas < 1.5, use the ``pandas.to_timedelta`` function instead:
 
     >>> df.play_time = pd.to_timedelta(df.play_time)
     >>> df.info()
diff --git a/requirements-dev.txt b/requirements-dev.txt
index d513016..17d68dd 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -4,6 +4,7 @@ flake8==3.8.4
 interrogate
 isort==5.12.0
 moto
+packaging
 pre-commit
 pytest
 pytest-cov


Test patch:
diff --git a/requirements-test.txt b/requirements-test.txt
index adb7955..4b4fee1 100644
--- a/requirements-test.txt
+++ b/requirements-test.txt
@@ -1,4 +1,5 @@
 moto[dynamodb]>=5,<6
+packaging
 pytest
 pytest-cov
 pytest-randomly
diff --git a/tests/test_main.py b/tests/test_main.py
index bccc3fe..e807209 100644
--- a/tests/test_main.py
+++ b/tests/test_main.py
@@ -1,6 +1,7 @@
 import re
 from unittest import mock
 
+from packaging.version import parse as parse_version
 import pandas as pd
 import pytest
 from test_data import test_df
@@ -166,12 +167,16 @@ def test_attributes_and_keys(self, test_df_table):
             )
         )
 
+    @pytest.mark.skipif(
+        parse_version(pd.__version__) < parse_version("1.5"),
+        reason="https://github.com/DrGFreeman/dynamo-pandas/issues/24",
+    )
     def test_dtype(self, test_df_table):
         """Test that the dtype parameter controls the returned data types."""
         df = get_df(
             table=test_df_table,
             dtype={
-                # "C": "timedelta64[ns]",  # Ref. #24
+                "C": "timedelta64[ns]",
                 "D": "datetime64[ns]",
                 "E": "datetime64[ns, UTC]",
                 "F": "Int32",
@@ -180,7 +185,7 @@ def test_dtype(self, test_df_table):
         assert {c: t.name for c, t in zip(df.columns, df.dtypes)} == {
             "A": "object",
             "B": "int64",
-            "C": "object",
+            "C": "timedelta64[ns]",
             "D": "datetime64[ns]",
             "E": "datetime64[ns, UTC]",
             "F": "Int32",
@@ -194,11 +199,7 @@ def test_boto3_kwargs_are_passed(self, ddb_client, test_df_table, keys):
         call."""
         # test_df_table is defined in us-east-1. By setting the region_name to
         # ca-central-1, we expect a ResourceNotFoundException.
-        # Moto raises a ValueError instead so expect it as well
-        # (see https://github.com/spulec/moto/issues/4344)
-        with pytest.raises(
-            (ddb_client.exceptions.ResourceNotFoundException, ValueError)
-        ):
+        with pytest.raises(ddb_client.exceptions.ResourceNotFoundException):
             get_df(
                 keys=keys,
                 table=test_df_table,
@@ -306,12 +307,16 @@ def test_multiple_item_dicts(self):
             "int64",
         ]
 
+    @pytest.mark.skipif(
+        parse_version(pd.__version__) < parse_version("1.5"),
+        reason="https://github.com/DrGFreeman/dynamo-pandas/issues/24",
+    )
     def test_with_dtype(self):
         """Test with dtype parameter specified."""
         df = _to_df(
             test_items,
             dtype=dict(
-                # C="timedelta64[ns]",  # Ref. #24
+                C="timedelta64[ns]",
                 D="datetime64[ns]",
                 E="datetime64[ns, UTC]",
                 F="Int32",
@@ -321,7 +326,7 @@ def test_with_dtype(self):
         assert [t.name for t in df.dtypes] == [
             "object",
             "int64",
-            "object",
+            "timedelta64[ns]",
             "datetime64[ns]",
             "datetime64[ns, UTC]",
             "Int32",