Repository: googleapis/python-spanner-django

Issue:
django: bitrightshift operator gives incorrect result on negative numbers
Failure:
```
======================================================================
FAIL: test_lefthand_bitwise_right_shift_operator (expressions.tests.ExpressionOperatorTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tim/code/django/tests/expressions/tests.py", line 1002, in test_lefthand_bitwise_right_shift_operator
    self.assertEqual(Number.objects.get(pk=self.n1.pk).integer, -11)
AssertionError: 4611686018427387893 != -11
```
Django commit: https://github.com/django/django/commit/1c12df4aa6c599959d9eb6de2076bf8aa6e301d6


Discussion:
The computation is `-42 >> 2` (expected -11, Spanner gave 4611686018427387893). Spanner's docs says this about right shift, "This operator does not do sign bit extension with a signed type (i.e. it fills vacant bits on the left with 0)." I'm not familiar enough with bit shift operators to say if there's an alternative computation we could implement within Django to give the expected result.
It looks like MySQL may behave the same as Spanner. In Django's MySQL backend, an alternate computation is used for right shift: `'FLOOR(%(lhs)s / POW(2, %(rhs)s))'`. Using that with Spanner may work except for a type mismatch issue: `Value of type FLOAT64 cannot be assigned to the_integer, which has type INT64 [at 1:46]\nUPDATE expressions_number SET the_integer = (FLOOR(expressions_number.the_int...\n`.
Interesting! So the result of that operation for `FLOOR(%(lhs)s / POW(2, %(rhs)s)` I believe returns a wrong result unless the goal is to round? With values like [-1001, 1001] >> 2 will result in [-251, 251] instead of [-250, 250] respectively. I think we'll need to instead use CEIL.

If the problem is that failed cast, here is how we can perhaps cast to INT64 and retrieve the correct result for both negative and positive
```python
return 'CAST(CEIL(%(lhs)s / POW(2, %(rhs)s)) AS INT64)`
```
The type mismatch issue is a Spanner limitation which I'm not sure if Django can solve (see also #331, for example). In this test the result column is int64, but it could be a float in another case in which case an integer cast will be incorrect.
@timgraham if there is a way for us to determine in Django the types of the left hand size or the result, we can perhaps issue different statements.
However, if that's impossible we might be fried here, and perhaps might have to hack around by trying different coercions depending on the exception encountered.

@bvandiver is there a way for us to specify a conditional in Google SQL to extract the output column's type?
@timgraham I've examined setting the result of INT64 in that case for columns of both FLOAT64 and INT64 and it doesn't crash:

### DDL
```sql
CREATE TABLE TestBitSet (
    id INT64,
    f FLOAT64,
    i INT64,
    s STRING(MAX),
) PRIMARY KEY (id)
```

### Experiment
#### Set INT64 result to FLOAT64 column
```sql
UPDATE TestBitSet AS ts SET ts.f = (CAST(CEIL(ts.i / POW(2, 3)) AS INT64)) where ts.id=1
```

#### Set INT64 result to INT64 column
```sql
UPDATE TestBitSet AS ts SET ts.t = (CAST(CEIL(ts.i / POW(2, 3)) AS INT64)) where ts.id=1
```

So, perhaps my suggestion in https://github.com/orijtech/django-spanner/issues/335#issuecomment-599288070 perhaps will work.

I now think we are no longer blocked.
I should have finished up with wording that

> float in another case in which case an integer cast will be incorrect.

Bit shifts on Cloud Spanner are intended for integer values or bytes and not float, so the usage of such in the first place of bitshfits for floats seems like a wrong use case at least for Cloud Spanner.
And for a correction, the equation used by MySQL and as @timgraham reported in https://github.com/orijtech/django-spanner/issues/335#issuecomment-599281952 is correct, mine is WRONG, it should use FLOOR(...), so that FLOOR(-250.25) gives a result of -251 and FLOOR(250.25) gives a result of 250.

Resolving patch:
diff --git a/django_spanner/features.py b/django_spanner/features.py
--- a/django_spanner/features.py
+++ b/django_spanner/features.py
@@ -160,9 +160,6 @@ class DatabaseFeatures(BaseDatabaseFeatures):
         'expressions.tests.FTimeDeltaTests.test_delta_add',
         'expressions.tests.FTimeDeltaTests.test_duration_with_datetime',
         'expressions.tests.FTimeDeltaTests.test_mixed_comparisons2',
-        # bitrightshift operator gives incorrect result:
-        # https://github.com/orijtech/django-spanner/issues/335
-        'expressions.tests.ExpressionOperatorTests.test_lefthand_bitwise_right_shift_operator',
         # integer division produces a float result, which can't be assigned to
         # an integer column:
         # https://github.com/orijtech/django-spanner/issues/331
diff --git a/django_spanner/operations.py b/django_spanner/operations.py
--- a/django_spanner/operations.py
+++ b/django_spanner/operations.py
@@ -231,6 +231,14 @@ def combine_expression(self, connector, sub_expressions):
             return 'MOD(%s)' % ', '.join(sub_expressions)
         elif connector == '^':
             return 'POWER(%s)' % ', '.join(sub_expressions)
+        elif connector == '>>':
+            lhs, rhs = sub_expressions
+            # Use an alternate computation because Cloud Sapnner's '>>' operator does not do
+            # sign bit extension with a signed type (i.e. produces different results for
+            # negative numbers than what Django's tests expect). Cast float result as INT64 to
+            # allow assigning to both INT64 and FLOAT64 columns (otherwise the FLOAT result
+            # couldn't be assigned to INT64 columns).
+            return 'CAST(FLOOR(%(lhs)s / POW(2, %(rhs)s)) AS INT64)' % {'lhs': lhs, 'rhs': rhs}
         return super().combine_expression(connector, sub_expressions)
 
     def combine_duration_expression(self, connector, sub_expressions):


Test patch: