Repository: ibis-project/ibis

Issue:
bug(bigquery): `TIMESTAMP` is unconditionally used for timestamp literals, even when lacking a timezone
### What happened?

In BigQuery connector, the final SQL output always cast to TIMETSAMP regardless of the given value type. 

For example: `df `is a Ibis table connecting to a table in BigQuery. The column `col_A `is `DATETIME `in BigQuery schema. Regardless the value input, the output SQL query are always `TIMESTAMP '2023-01-01'`..... However, the first one should `DATETIME`, while the second one is `TIMESTAMP`

`df.col_A.fillna(ibis.literal("2023-01-01", type="timestamp")).compile()`
`"SELECT IFNULL(t0.`col_A`, TIMESTAMP '2023-01-01 00:00:00') AS `IfNull_col_A_ datetime_datetime_2023_ 1_ 1_ 0_ 0`\nFROM `beta-385421.test_integration_eric_1695754492_2081.table_na_datetimes` t0"`

`df.col_A.fillna(ibis.literal("2023-01-01", type="timestamp('UTC')")).compile()`
`"SELECT IFNULL(t0.`col_A`, TIMESTAMP '2023-01-01 00:00:00+00:00') AS `IfNull_col_A_ datetime_datetime_2023_ 1_ 1_ 0_ 0_ tzinfo=tzutc`\nFROM `beta-385421.test_integration_eric_1695754492_2081.table_na_datetimes` t0"`



### What version of ibis are you using?

6

### What backend(s) are you using, if any?

_No response_

### Relevant log output

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
Thanks the report, I can confirm we are hardcoding the use of `TIMESTAMP` for literal compilation.
That explains the issue. But will this be changed in the future, since the output SQL query would break if the target column is be filled is not `TIMESTAMP`?
We're already treating `timestamp` types without a time zone as `datetime`s for the bigquery backend, this was just a missed changed from the PR that made those changes (https://github.com/ibis-project/ibis/pull/6144)
There is no released version that has a fix for this issue. #6144 did not have changes corresponding to `literal` compilation which is the problem you're seeing.

Good news is you opened this right before our 7.0.0 release which is slated for next Monday!

Resolving patch:
diff --git a/ibis/backends/bigquery/registry.py b/ibis/backends/bigquery/registry.py
--- a/ibis/backends/bigquery/registry.py
+++ b/ibis/backends/bigquery/registry.py
@@ -362,7 +362,8 @@ def _literal(t, op):
             value = value.date()
         return f"DATE {_sg_literal(str(value))}"
     elif dtype.is_timestamp():
-        return f"TIMESTAMP {_sg_literal(str(value))}"
+        typename = "DATETIME" if dtype.timezone is None else "TIMESTAMP"
+        return f"{typename} {_sg_literal(str(value))}"
     elif dtype.is_time():
         # TODO: define extractors on TimeValue expressions
         return f"TIME {_sg_literal(str(value))}"


Test patch:
diff --git a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/datetime/index.sql b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/datetime/index.sql
--- a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/datetime/index.sql
+++ b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/datetime/index.sql
@@ -1,2 +1,2 @@
 SELECT
-  MOD(EXTRACT(DAYOFWEEK FROM CAST('2017-01-01 04:55:59' AS TIMESTAMP)) + 5, 7) AS `DayOfWeekIndex_datetime_datetime_2017_ 1_ 1_ 4_ 55_ 59`
\ No newline at end of file
+  MOD(EXTRACT(DAYOFWEEK FROM CAST('2017-01-01 04:55:59' AS DATETIME)) + 5, 7) AS `DayOfWeekIndex_datetime_datetime_2017_ 1_ 1_ 4_ 55_ 59`
\ No newline at end of file
diff --git a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/datetime/name.sql b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/datetime/name.sql
--- a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/datetime/name.sql
+++ b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/datetime/name.sql
@@ -1,2 +1,2 @@
 SELECT
-  INITCAP(CAST(CAST('2017-01-01 04:55:59' AS TIMESTAMP) AS STRING FORMAT 'DAY')) AS `DayOfWeekName_datetime_datetime_2017_ 1_ 1_ 4_ 55_ 59`
\ No newline at end of file
+  INITCAP(CAST(CAST('2017-01-01 04:55:59' AS DATETIME) AS STRING FORMAT 'DAY')) AS `DayOfWeekName_datetime_datetime_2017_ 1_ 1_ 4_ 55_ 59`
\ No newline at end of file
diff --git a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/string_timestamp/index.sql b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/string_timestamp/index.sql
--- a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/string_timestamp/index.sql
+++ b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/string_timestamp/index.sql
@@ -1,2 +1,2 @@
 SELECT
-  MOD(EXTRACT(DAYOFWEEK FROM CAST('2017-01-01 04:55:59' AS TIMESTAMP)) + 5, 7) AS `DayOfWeekIndex_datetime_datetime_2017_ 1_ 1_ 4_ 55_ 59`
\ No newline at end of file
+  MOD(EXTRACT(DAYOFWEEK FROM CAST('2017-01-01 04:55:59' AS DATETIME)) + 5, 7) AS `DayOfWeekIndex_datetime_datetime_2017_ 1_ 1_ 4_ 55_ 59`
\ No newline at end of file
diff --git a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/string_timestamp/name.sql b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/string_timestamp/name.sql
--- a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/string_timestamp/name.sql
+++ b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/string_timestamp/name.sql
@@ -1,2 +1,2 @@
 SELECT
-  INITCAP(CAST(CAST('2017-01-01 04:55:59' AS TIMESTAMP) AS STRING FORMAT 'DAY')) AS `DayOfWeekName_datetime_datetime_2017_ 1_ 1_ 4_ 55_ 59`
\ No newline at end of file
+  INITCAP(CAST(CAST('2017-01-01 04:55:59' AS DATETIME) AS STRING FORMAT 'DAY')) AS `DayOfWeekName_datetime_datetime_2017_ 1_ 1_ 4_ 55_ 59`
\ No newline at end of file
diff --git a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/timestamp/index.sql b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/timestamp/index.sql
--- a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/timestamp/index.sql
+++ b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/timestamp/index.sql
@@ -1,2 +1,2 @@
 SELECT
-  MOD(EXTRACT(DAYOFWEEK FROM CAST('2017-01-01 04:55:59' AS TIMESTAMP)) + 5, 7) AS `DayOfWeekIndex_datetime_datetime_2017_ 1_ 1_ 4_ 55_ 59`
\ No newline at end of file
+  MOD(EXTRACT(DAYOFWEEK FROM CAST('2017-01-01 04:55:59' AS DATETIME)) + 5, 7) AS `DayOfWeekIndex_datetime_datetime_2017_ 1_ 1_ 4_ 55_ 59`
\ No newline at end of file
diff --git a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/timestamp/name.sql b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/timestamp/name.sql
--- a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/timestamp/name.sql
+++ b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_day_of_week/timestamp/name.sql
@@ -1,2 +1,2 @@
 SELECT
-  INITCAP(CAST(CAST('2017-01-01 04:55:59' AS TIMESTAMP) AS STRING FORMAT 'DAY')) AS `DayOfWeekName_datetime_datetime_2017_ 1_ 1_ 4_ 55_ 59`
\ No newline at end of file
+  INITCAP(CAST(CAST('2017-01-01 04:55:59' AS DATETIME) AS STRING FORMAT 'DAY')) AS `DayOfWeekName_datetime_datetime_2017_ 1_ 1_ 4_ 55_ 59`
\ No newline at end of file
diff --git a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_timestamp_or_time/datetime/out.sql b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_timestamp_or_time/datetime/out.sql
--- a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_timestamp_or_time/datetime/out.sql
+++ b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_timestamp_or_time/datetime/out.sql
@@ -1,2 +1,2 @@
 SELECT
-  EXTRACT(hour FROM CAST('2017-01-01 04:55:59' AS TIMESTAMP)) AS `tmp`
\ No newline at end of file
+  EXTRACT(hour FROM CAST('2017-01-01 04:55:59' AS DATETIME)) AS `tmp`
\ No newline at end of file
diff --git a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_timestamp_or_time/string_timestamp/out.sql b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_timestamp_or_time/string_timestamp/out.sql
--- a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_timestamp_or_time/string_timestamp/out.sql
+++ b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_timestamp_or_time/string_timestamp/out.sql
@@ -1,2 +1,2 @@
 SELECT
-  EXTRACT(hour FROM CAST('2017-01-01 04:55:59' AS TIMESTAMP)) AS `tmp`
\ No newline at end of file
+  EXTRACT(hour FROM CAST('2017-01-01 04:55:59' AS DATETIME)) AS `tmp`
\ No newline at end of file
diff --git a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_timestamp_or_time/timestamp/out.sql b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_timestamp_or_time/timestamp/out.sql
--- a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_timestamp_or_time/timestamp/out.sql
+++ b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_timestamp_or_time/timestamp/out.sql
@@ -1,2 +1,2 @@
 SELECT
-  EXTRACT(hour FROM CAST('2017-01-01 04:55:59' AS TIMESTAMP)) AS `tmp`
\ No newline at end of file
+  EXTRACT(hour FROM CAST('2017-01-01 04:55:59' AS DATETIME)) AS `tmp`
\ No newline at end of file
diff --git a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_year/datetime/out.sql b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_year/datetime/out.sql
--- a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_year/datetime/out.sql
+++ b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_year/datetime/out.sql
@@ -1,2 +1,2 @@
 SELECT
-  EXTRACT(year FROM CAST('2017-01-01 04:55:59' AS TIMESTAMP)) AS `ExtractYear_datetime_datetime_2017_ 1_ 1_ 4_ 55_ 59`
\ No newline at end of file
+  EXTRACT(year FROM CAST('2017-01-01 04:55:59' AS DATETIME)) AS `ExtractYear_datetime_datetime_2017_ 1_ 1_ 4_ 55_ 59`
\ No newline at end of file
diff --git a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_year/string_timestamp/out.sql b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_year/string_timestamp/out.sql
--- a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_year/string_timestamp/out.sql
+++ b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_year/string_timestamp/out.sql
@@ -1,2 +1,2 @@
 SELECT
-  EXTRACT(year FROM CAST('2017-01-01 04:55:59' AS TIMESTAMP)) AS `ExtractYear_datetime_datetime_2017_ 1_ 1_ 4_ 55_ 59`
\ No newline at end of file
+  EXTRACT(year FROM CAST('2017-01-01 04:55:59' AS DATETIME)) AS `ExtractYear_datetime_datetime_2017_ 1_ 1_ 4_ 55_ 59`
\ No newline at end of file
diff --git a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_year/timestamp/out.sql b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_year/timestamp/out.sql
--- a/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_year/timestamp/out.sql
+++ b/ibis/backends/bigquery/tests/unit/snapshots/test_compiler/test_literal_year/timestamp/out.sql
@@ -1,2 +1,2 @@
 SELECT
-  EXTRACT(year FROM CAST('2017-01-01 04:55:59' AS TIMESTAMP)) AS `ExtractYear_datetime_datetime_2017_ 1_ 1_ 4_ 55_ 59`
\ No newline at end of file
+  EXTRACT(year FROM CAST('2017-01-01 04:55:59' AS DATETIME)) AS `ExtractYear_datetime_datetime_2017_ 1_ 1_ 4_ 55_ 59`
\ No newline at end of file
diff --git a/ibis/backends/tests/test_temporal.py b/ibis/backends/tests/test_temporal.py
--- a/ibis/backends/tests/test_temporal.py
+++ b/ibis/backends/tests/test_temporal.py
@@ -1258,6 +1258,12 @@ def test_timestamp_comparison_filter(backend, con, alltypes, df, func_name):
     raises=BaseException,  # pyo3_runtime.PanicException is not a subclass of Exception
     reason="failed to determine supertype of datetime[ns, UTC] and datetime[ns]",
 )
+@pytest.mark.never(
+    ["bigquery"],
+    raises=GoogleBadRequest,
+    # perhaps we should consider disallowing this in ibis as well
+    reason="BigQuery doesn't support comparing DATETIME and TIMESTAMP; numpy doesn't support timezones",
+)
 def test_timestamp_comparison_filter_numpy(backend, con, alltypes, df, func_name):
     ts = np.datetime64("2010-03-02 00:00:00.000123")
 
@@ -2215,11 +2221,6 @@ def test_integer_cast_to_timestamp_scalar(alltypes, df):
     reason="PySpark doesn't handle big timestamps",
     raises=pd.errors.OutOfBoundsDatetime,
 )
-@pytest.mark.notimpl(
-    ["bigquery"],
-    reason="bigquery returns a datetime with a timezone",
-    raises=AssertionError,
-)
 def test_big_timestamp(con):
     # TODO: test with a timezone
     value = ibis.timestamp("2419-10-11 10:10:25")