Repository: googleapis/python-spanner-django

Issue:
django: can't use QuerySet.dates() on DateTimeField
`QuerySet.dates()` calls `DatabaseOperations.date_trunc_sql()` which uses `DATE_TRUNC`, however, `DATE_TRUNC()` can't be used on `Timestamp` (`DateTimeField`).
```
======================================================================
ERROR: test_dates_trunc_datetime_fields (dates.tests.DatesTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tim/.virtualenvs/django37/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 79, in next
    return six.next(self._wrapped)
  File "/home/tim/.virtualenvs/django37/lib/python3.7/site-packages/grpc/_channel.py", line 392, in __next__
    return self._next()
  File "/home/tim/.virtualenvs/django37/lib/python3.7/site-packages/grpc/_channel.py", line 561, in _next
    raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "No matching signature for function DATE_TRUNC for argument types: TIMESTAMP, DATE_TIME_PART. Supported signature: DATE_TRUNC(DATE, DATE_TIME_PART) [at 1:17]\nSELECT DISTINCT DATE_TRUNC(dates_article.pub_datetime, day) AS datefield FROM...\n                ^"
	debug_error_string = "{"created":"@1576193555.291582490","description":"Error received from peer ipv4:172.217.10.106:443","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"No matching signature for function DATE_TRUNC for argument types: TIMESTAMP, DATE_TIME_PART. Supported signature: DATE_TRUNC(DATE, DATE_TIME_PART) [at 1:17]\nSELECT DISTINCT DATE_TRUNC(dates_article.pub_datetime, day) AS datefield FROM...\n                ^","grpc_status":3}"
```
It might be possible to first cast the timestamp to a date in SQL.


Discussion:
Also seen from queryset_pickle.tests.PickleabilityTestCase.test_specialized_queryset
```shell
======================================================================
ERROR: test_specialized_queryset (queryset_pickle.tests.PickleabilityTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 79, in next
    return six.next(self._wrapped)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grpc/_channel.py", line 364, in __next__
    return self._next()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grpc/_channel.py", line 358, in _next
    raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "No matching signature for function DATE_TRUNC for argument types: TIMESTAMP, DATE_TIME_PART. Supported signature: DATE_TRUNC(DATE, DATE_TIME_PART) [at 1:17]\nSELECT DISTINCT DATE_TRUNC(queryset_pickle_happening.`when`, year) AS datefie...\n                ^"
	debug_error_string = "{"created":"@1583220456.759889000","description":"Error received from peer ipv6:[2607:f8b0:4005:805::200a]:443","file":"src/core/lib/surface/call.cc","file_line":1046,"grpc_message":"No matching signature for function DATE_TRUNC for argument types: TIMESTAMP, DATE_TIME_PART. Supported signature: DATE_TRUNC(DATE, DATE_TIME_PART) [at 1:17]\nSELECT DISTINCT DATE_TRUNC(queryset_pickle_happening.`when`, year) AS datefie...\n                ^","grpc_status":3}"
>
```
I just tickled this while testing out the fix for #262. The solution to such a problem would perhaps
```sql
DATE_TRUNC(DATE(db_functions_dtmodel.start_datetime, 'UTC'), day)
```
but the problem here is that some fields might be passed in, of type DATE and then that conversion fails.

Given that the signature of the respective *date*_trunc methods is quite limiting i.e.
```python
def date_trunc_sql(self, lookup_type, field_name):
     pass
```
We perhaps need a way to infer types of the various field names if possible.
@timgraham is there perhaps an object in DatabaseOperations that let's us map the various model field names to their respective types, and if not, can we perhaps make a feature request for Django? The need for this feature has come up time and again on a bunch of open issues on this project.
Yeah, as I've said before, I'd like to hear from Spanner engineers about this and the similar issues you mentioned. It doesn't seem solvable in Django and I would imagine other frameworks have run into these limitations.
@timgraham, could you please craft up the question and I'll get us on an email thread with the Spanner engineering team?
While working on the Django engine for Spanner, we've encountered an issue for which we don't see a solution.

Consider this method from Spanner's Django engine:

class DatabaseOperations:
    def date_trunc_sql(self, lookup_type, field_name):
        return 'DATE_TRUNC(%s, %s)' % (field_name, lookup_type)

field_name is a string like 'table.column' and lookup_type is something like 'year'. 

It works if the field is a DATE, but fails if the field is a TIMESTAMP. Spanner's TIMESTAMP_TRUNC() function has to be used instead.

Solutions we tried:
1. Use a DATE cast like this:
DATE_TRUNC(DATE(db_functions_dtmodel.start_datetime, 'UTC'), day)
It works if the column is a TIMESTAMP but fails in when the column is a DATE (DATE cannot be cast to itself).
2. Build a mapping of 'table.column' to field types that the method could use to choose DATE_TRUNC or TIMESTAMP_TRUNC appropriately. The problem is that the mapping isn't possible to do for more complex queries where table.column is an alias.
This is a very good question but there's no simple solution.

Based on what's available, the only solution I could think of that would be common to both types is `EXTRACT` ([timestamp EXTRACT](https://cloud.google.com/spanner/docs/functions-and-operators#extract_2), [date EXTRACT](https://cloud.google.com/spanner/docs/functions-and-operators#extract)) but this means you're implementing the truncate yourself.

The other option is to join with an information_schema query but I don't even know if that's possible and is still messy.

The cleanest solutions would have been the DATE cast. Are you sure that didn't work? I tried it at and it seems to work fine. Can you share the error that you get?

![Screenshot 2020-03-24 at 11 15 17 AM](https://user-images.githubusercontent.com/951152/77375170-d0a76080-6dc0-11ea-9f0e-64ea5acdea0a.png)


Resolving patch:
diff --git a/django_spanner/features.py b/django_spanner/features.py
--- a/django_spanner/features.py
+++ b/django_spanner/features.py
@@ -163,11 +163,6 @@ class DatabaseFeatures(BaseDatabaseFeatures):
         'syndication_tests.tests.SyndicationFeedTest.test_latest_post_date',
         'syndication_tests.tests.SyndicationFeedTest.test_rss091_feed',
         'syndication_tests.tests.SyndicationFeedTest.test_template_feed',
-        # DATE_TRUNC cannot be used on DateTimeField:
-        # https://github.com/orijtech/spanner-orm/issues/182
-        'backends.tests.DateQuotingTest.test_django_date_trunc',
-        'dates.tests.DatesTests.test_dates_trunc_datetime_fields',
-        'db_functions.datetime.test_extract_trunc.DateFunctionTests.test_trunc_func',
         # datetimes retrieved from the database with the wrong hour when
         # USE_TZ = True: https://github.com/orijtech/spanner-orm/issues/193
         'datetimes.tests.DateTimesTests.test_21432',
diff --git a/django_spanner/operations.py b/django_spanner/operations.py
--- a/django_spanner/operations.py
+++ b/django_spanner/operations.py
@@ -184,11 +184,11 @@ def date_trunc_sql(self, lookup_type, field_name):
             # Spanner truncates to Sunday but Django expects Monday. First,
             # subtract a day so that a Sunday will be truncated to the previous
             # week...
-            field_name = 'DATE_SUB(' + field_name + ', INTERVAL 1 DAY)'
-        sql = 'DATE_TRUNC(%s, %s)' % (field_name, lookup_type)
+            field_name = 'DATE_SUB(CAST(' + field_name + ' AS DATE), INTERVAL 1 DAY)'
+        sql = 'DATE_TRUNC(CAST(%s AS DATE), %s)' % (field_name, lookup_type)
         if lookup_type == 'week':
             # ...then add a day to get from Sunday to Monday.
-            sql = 'DATE_ADD(' + sql + ', INTERVAL 1 DAY)'
+            sql = 'DATE_ADD(CAST(' + sql + ' AS DATE), INTERVAL 1 DAY)'
         return sql
 
     def datetime_trunc_sql(self, lookup_type, field_name, tzname):


Test patch: