Repository: getmoto/moto
Pull request #5915 (2023-02-09 18:03:41)

Glue: support nanosecond-precision timestamp partition filtering
This PR supports nanosecond-resolution timestamp partition filters for Glue. It does so by, instead of using a Python `datetime` to compare timestamps, using POSIX timestamps augmented to nanosecond resolution.

The validity of the timestamp string is still checked, but the POSIX representation is used internally for comparisons.

robert-schmidtke: Will fix the import lint tomorrow!
codecov[bot]: # [Codecov](https://codecov.io/gh/getmoto/moto/pull/5915?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) Report
> Merging [#5915](https://codecov.io/gh/getmoto/moto/pull/5915?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) (2cc8dd5) into [master](https://codecov.io/gh/getmoto/moto/commit/c6c0e50ee9415df4aab76d799978721771b87afd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) (c6c0e50) will **not change** coverage.
> The diff coverage is `100.00%`.

```diff
@@           Coverage Diff           @@
##           master    #5915   +/-   ##
=======================================
  Coverage   96.22%   96.22%           
=======================================
  Files         759      759           
  Lines       74793    74793           
=======================================
  Hits        71967    71967           
  Misses       2826     2826           
```

| Flag | Coverage Δ | |
|---|---|---|
| servertests | `36.95% <25.00%> (ø)` | |
| unittests | `96.16% <100.00%> (ø)` | |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/getmoto/moto/pull/5915?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) | Coverage Δ | |
|---|---|---|
| [moto/glue/models.py](https://codecov.io/gh/getmoto/moto/pull/5915?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-bW90by9nbHVlL21vZGVscy5weQ==) | `97.13% <ø> (ø)` | |
| [moto/glue/utils.py](https://codecov.io/gh/getmoto/moto/pull/5915?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-bW90by9nbHVlL3V0aWxzLnB5) | `100.00% <100.00%> (ø)` | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None)


Patch:
diff --git a/moto/glue/models.py b/moto/glue/models.py
index 0152b8b67..31f3bd1be 100644
--- a/moto/glue/models.py
+++ b/moto/glue/models.py
@@ -173,7 +173,6 @@ class GlueBackend(BaseBackend):
         Expression caveats:
 
         - Column names must consist of UPPERCASE, lowercase, dots and underscores only.
-        - Nanosecond expressions on timestamp columns are rounded to microseconds.
         - Literal dates and timestamps must be valid, i.e. no support for February 31st.
         - LIKE expressions are converted to Python regexes, escaping special characters.
           Only % and _ wildcards are supported, and SQL escaping using [] does not work.
diff --git a/moto/glue/utils.py b/moto/glue/utils.py
index 1e3600fb5..4b1f45003 100644
--- a/moto/glue/utils.py
+++ b/moto/glue/utils.py
@@ -2,7 +2,7 @@ import abc
 import operator
 import re
 import warnings
-from datetime import date, datetime, timedelta
+from datetime import date, datetime
 from itertools import repeat
 from typing import Any, Dict, List, Optional, Union
 
@@ -74,15 +74,17 @@ def _cast(type_: str, value: Any) -> Union[date, datetime, float, int, str]:
                 f" {value} is not a timestamp."
             )
 
+        # use nanosecond representation for timestamps
+        posix_nanoseconds = int(timestamp.timestamp() * 1_000_000_000)
+
         nanos = match.group("nanos")
         if nanos is not None:
             # strip leading dot, reverse and left pad with zeros to nanoseconds
             nanos = "".join(reversed(nanos[1:])).zfill(9)
             for i, nanoseconds in enumerate(nanos):
-                microseconds = (int(nanoseconds) * 10**i) / 1000
-                timestamp += timedelta(microseconds=round(microseconds))
+                posix_nanoseconds += int(nanoseconds) * 10**i
 
-        return timestamp
+        return posix_nanoseconds
 
     raise InvalidInputException("GetPartitions", f"Unknown type : '{type_}'")
 


Test patch:
diff --git a/tests/test_glue/test_partition_filter.py b/tests/test_glue/test_partition_filter.py
index b1b70c6f0..4bfa6aa1b 100644
--- a/tests/test_glue/test_partition_filter.py
+++ b/tests/test_glue/test_partition_filter.py
@@ -288,11 +288,10 @@ def test_get_partitions_expression_timestamp_column():
         "timestamp_col between '2022-01-15 00:00:00' AND '2022-02-15 00:00:00'",
         "timestamp_col > '2022-01-15 00:00:00' AND "
         "timestamp_col < '2022-02-15 00:00:00'",
-        # these expressions only work because of rounding to microseconds
-        "timestamp_col = '2022-01-31 23:59:59.999999999'",
-        "timestamp_col = '2022-02-01 00:00:00.00000001'",
         "timestamp_col > '2022-01-31 23:59:59.999999499' AND"
         " timestamp_col < '2022-02-01 00:00:00.0000009'",
+        "timestamp_col > '2022-01-31 23:59:59.999999999' AND"
+        " timestamp_col < '2022-02-01 00:00:00.000000001'",
     )
 
     for expression in timestamp_col_is_february_expressions: