Repository: googleapis/python-bigquery

Issue:
Use DataFormatOptions to download timestamps with full precision
 **Is your feature request related to a problem? Please describe.**

When fetching timestamp values, the [tabledata.list](https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/list) and [jobs.getQueryResults](https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/getQueryResults) APIs return a floating point value by default. 

This can cause [unexpected floating point issues](https://issuetracker.google.com/issues/147629356) when rounding timestamps in SQL, such as with the `TIMESTAMP_TRUNC(COL_TIMESTAMP, MILLISECOND)` function.

 **Describe the solution you'd like**

The backend API adds a [`formatOptions.useInt64Timestamp`](https://cloud.google.com/bigquery/docs/reference/rest/v2/DataFormatOptions#FIELDS.use_int64_timestamp) option. 

* Wherever `tabledata.list`, `jobs.query`, or `jobs.getQueryResults` are called, set the `formatOptions.useInt64Timestamp` option to `True`.
* Wherever timestamps in rows API responses are handled, update it to use integer math to extract microseconds and seconds.
* Add system tests to verify that timestamps rounded to every digit of supported precision (e.g. 9, 9.9, 9.99, ..., 9.999999) are returned exactly as set in the table/query.

 **Describe alternatives you've considered**

N/A

 **Additional context**

Precision issues for the other direction (to BigQuery) for similar floating-point reasons: https://github.com/googleapis/python-bigquery/issues/392



Discussion:
@tswast Do we need anything in `.proto` file here because when i am passing in `query_paramter` it throws an error.
Why are you passing this to `query_parameter`? These are API parameters (called `extra_params` in our client code, I believe)
Yes , I am passing in extra_params like
extra_parms['formatOptions']=Dataformat options(object)

On Mon, 23 Nov, 2020, 9:01 pm Tim Swast, <notifications@github.com> wrote:

> Why are you passing this to query_parameter? These are API parameters
> (called extra_params in our client code, I believe)
>
> —
> You are receiving this because you were assigned.
> Reply to this email directly, view it on GitHub
> <https://github.com/googleapis/python-bigquery/issues/395#issuecomment-732234478>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AMAR4JWAJL2SI4ZFD24YKPTSRJ537ANCNFSM4T3QUAZA>
> .
>

When i pass the `extra_parms['formatOptions']=Dataformatoptions`  (object) as mentioned in [dcument](https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/list#query-parameters), it throws an error like  `Invalid JSON payload received. Unknown name "formatOptions": Cannot bind query parameter. 'formatOptions' is a message type. Parameters can only be bound to primitive types.`

https://github.com/googleapis/python-bigquery/blob/a1949ae20ec4f9c771b0cffbcd70792dd6a30dbf/google/cloud/bigquery/client.py#L3154-L3168



@HemangChothani  Please try the following:

* Does it work if you first serialize the DataFormatOption dictionary to JSON? e.g. `json.dumps({"useInt64Timestamp": True})`
* Does it work if instead of setting `extra_parms['formatOptions']`, you set `extra_parms['formatOptions.useInt64Timestamp']`?

Resolving patch:
diff --git a/google/cloud/bigquery/_helpers.py b/google/cloud/bigquery/_helpers.py
--- a/google/cloud/bigquery/_helpers.py
+++ b/google/cloud/bigquery/_helpers.py
@@ -81,8 +81,8 @@ def _bytes_from_json(value, field):
 def _timestamp_from_json(value, field):
     """Coerce 'value' to a datetime, if set or not nullable."""
     if _not_null(value, field):
-        # value will be a float in seconds, to microsecond precision, in UTC.
-        return _datetime_from_microseconds(1e6 * float(value))
+        # value will be a integer in seconds, to microsecond precision, in UTC.
+        return _datetime_from_microseconds(int(value))
 
 
 def _timestamp_query_param_from_json(value, field):
diff --git a/google/cloud/bigquery/client.py b/google/cloud/bigquery/client.py
--- a/google/cloud/bigquery/client.py
+++ b/google/cloud/bigquery/client.py
@@ -3157,6 +3157,7 @@ def list_rows(
         if start_index is not None:
             params["startIndex"] = start_index
 
+        params["formatOptions.useInt64Timestamp"] = True
         row_iterator = RowIterator(
             client=self,
             api_request=functools.partial(self._call_api, retry, timeout=timeout),
@@ -3237,6 +3238,7 @@ def _list_rows_from_query_results(
         if start_index is not None:
             params["startIndex"] = start_index
 
+        params["formatOptions.useInt64Timestamp"] = True
         row_iterator = RowIterator(
             client=self,
             api_request=functools.partial(self._call_api, retry, timeout=timeout),


Test patch:
diff --git a/tests/unit/job/test_query.py b/tests/unit/job/test_query.py
--- a/tests/unit/job/test_query.py
+++ b/tests/unit/job/test_query.py
@@ -839,6 +839,7 @@ def test_result(self):
             query_params={
                 "fields": _LIST_ROWS_FROM_QUERY_RESULTS_FIELDS,
                 "location": "EU",
+                "formatOptions.useInt64Timestamp": True,
             },
             timeout=None,
         )
@@ -887,6 +888,7 @@ def test_result_with_done_job_calls_get_query_results(self):
             query_params={
                 "fields": _LIST_ROWS_FROM_QUERY_RESULTS_FIELDS,
                 "location": "EU",
+                "formatOptions.useInt64Timestamp": True,
             },
             timeout=None,
         )
@@ -1118,6 +1120,7 @@ def test_result_w_page_size(self):
                 "maxResults": 3,
                 "fields": _LIST_ROWS_FROM_QUERY_RESULTS_FIELDS,
                 "location": "US",
+                "formatOptions.useInt64Timestamp": True,
             },
             timeout=None,
         )
@@ -1129,6 +1132,7 @@ def test_result_w_page_size(self):
                 "maxResults": 3,
                 "fields": _LIST_ROWS_FROM_QUERY_RESULTS_FIELDS,
                 "location": "US",
+                "formatOptions.useInt64Timestamp": True,
             },
             timeout=None,
         )
diff --git a/tests/unit/job/test_query_pandas.py b/tests/unit/job/test_query_pandas.py
--- a/tests/unit/job/test_query_pandas.py
+++ b/tests/unit/job/test_query_pandas.py
@@ -501,7 +501,7 @@ def test_to_dataframe_column_dtypes():
     }
     row_data = [
         [
-            "1.4338368E9",
+            "1433836800000000",
             "420",
             "1.1",
             "1.77",
@@ -509,8 +509,8 @@ def test_to_dataframe_column_dtypes():
             "true",
             "1999-12-01",
         ],
-        ["1.3878117E9", "2580", "17.7", "28.5", "Cash", "false", "1953-06-14"],
-        ["1.3855653E9", "2280", "4.4", "7.1", "Credit", "true", "1981-11-04"],
+        ["1387811700000000", "2580", "17.7", "28.5", "Cash", "false", "1953-06-14"],
+        ["1385565300000000", "2280", "4.4", "7.1", "Credit", "true", "1981-11-04"],
     ]
     rows = [{"f": [{"v": field} for field in row]} for row in row_data]
     query_resource["rows"] = rows
diff --git a/tests/unit/test__helpers.py b/tests/unit/test__helpers.py
--- a/tests/unit/test__helpers.py
+++ b/tests/unit/test__helpers.py
@@ -190,18 +190,18 @@ def test_w_none_required(self):
         with self.assertRaises(TypeError):
             self._call_fut(None, _Field("REQUIRED"))
 
-    def test_w_string_value(self):
+    def test_w_string_int_value(self):
         from google.cloud._helpers import _EPOCH
 
-        coerced = self._call_fut("1.234567", object())
+        coerced = self._call_fut("1234567", object())
         self.assertEqual(
             coerced, _EPOCH + datetime.timedelta(seconds=1, microseconds=234567)
         )
 
-    def test_w_float_value(self):
+    def test_w_int_value(self):
         from google.cloud._helpers import _EPOCH
 
-        coerced = self._call_fut(1.234567, object())
+        coerced = self._call_fut(1234567, object())
         self.assertEqual(
             coerced, _EPOCH + datetime.timedelta(seconds=1, microseconds=234567)
         )
diff --git a/tests/unit/test_client.py b/tests/unit/test_client.py
--- a/tests/unit/test_client.py
+++ b/tests/unit/test_client.py
@@ -6739,42 +6739,21 @@ def test_list_rows(self):
             self.DS_ID,
             self.TABLE_ID,
         )
-        WHEN_TS = 1437767599.006
-        WHEN = datetime.datetime.utcfromtimestamp(WHEN_TS).replace(tzinfo=UTC)
-        WHEN_1 = WHEN + datetime.timedelta(seconds=1)
-        WHEN_2 = WHEN + datetime.timedelta(seconds=2)
+        WHEN_TS = 1437767599006000
+
+        WHEN = datetime.datetime.utcfromtimestamp(WHEN_TS / 1e6).replace(tzinfo=UTC)
+        WHEN_1 = WHEN + datetime.timedelta(microseconds=1)
+        WHEN_2 = WHEN + datetime.timedelta(microseconds=2)
         ROWS = 1234
         TOKEN = "TOKEN"
 
-        def _bigquery_timestamp_float_repr(ts_float):
-            # Preserve microsecond precision for E+09 timestamps
-            return "%0.15E" % (ts_float,)
-
         DATA = {
             "totalRows": str(ROWS),
             "pageToken": TOKEN,
             "rows": [
-                {
-                    "f": [
-                        {"v": "Phred Phlyntstone"},
-                        {"v": "32"},
-                        {"v": _bigquery_timestamp_float_repr(WHEN_TS)},
-                    ]
-                },
-                {
-                    "f": [
-                        {"v": "Bharney Rhubble"},
-                        {"v": "33"},
-                        {"v": _bigquery_timestamp_float_repr(WHEN_TS + 1)},
-                    ]
-                },
-                {
-                    "f": [
-                        {"v": "Wylma Phlyntstone"},
-                        {"v": "29"},
-                        {"v": _bigquery_timestamp_float_repr(WHEN_TS + 2)},
-                    ]
-                },
+                {"f": [{"v": "Phred Phlyntstone"}, {"v": "32"}, {"v": WHEN_TS}]},
+                {"f": [{"v": "Bharney Rhubble"}, {"v": "33"}, {"v": WHEN_TS + 1}]},
+                {"f": [{"v": "Wylma Phlyntstone"}, {"v": "29"}, {"v": WHEN_TS + 2}]},
                 {"f": [{"v": "Bhettye Rhubble"}, {"v": None}, {"v": None}]},
             ],
         }
@@ -6807,7 +6786,10 @@ def _bigquery_timestamp_float_repr(ts_float):
         self.assertEqual(iterator.next_page_token, TOKEN)
 
         conn.api_request.assert_called_once_with(
-            method="GET", path="/%s" % PATH, query_params={}, timeout=7.5
+            method="GET",
+            path="/%s" % PATH,
+            query_params={"formatOptions.useInt64Timestamp": True},
+            timeout=7.5,
         )
 
     def test_list_rows_w_start_index_w_page_size(self):
@@ -6856,20 +6838,30 @@ def test_list_rows_w_start_index_w_page_size(self):
         self.assertEqual(len(rows), 2)
         self.assertEqual(rows[0], Row(("Wylma Phlyntstone",), f2i))
         self.assertEqual(rows[1], Row(("Bhettye Rhubble",), f2i))
-        self.assertEqual(extra_params, {"startIndex": 1})
+        self.assertEqual(
+            extra_params, {"startIndex": 1, "formatOptions.useInt64Timestamp": True}
+        )
 
         conn.api_request.assert_has_calls(
             [
                 mock.call(
                     method="GET",
                     path="/%s" % PATH,
-                    query_params={"startIndex": 1, "maxResults": 2},
+                    query_params={
+                        "startIndex": 1,
+                        "maxResults": 2,
+                        "formatOptions.useInt64Timestamp": True,
+                    },
                     timeout=None,
                 ),
                 mock.call(
                     method="GET",
                     path="/%s" % PATH,
-                    query_params={"pageToken": "some-page-token", "maxResults": 2},
+                    query_params={
+                        "pageToken": "some-page-token",
+                        "maxResults": 2,
+                        "formatOptions.useInt64Timestamp": True,
+                    },
                     timeout=None,
                 ),
             ]
@@ -6920,6 +6912,7 @@ def test_list_rows_query_params(self):
             iterator = client.list_rows(table, **test[0])
             six.next(iterator.pages)
             req = conn.api_request.call_args_list[i]
+            test[1]["formatOptions.useInt64Timestamp"] = True
             self.assertEqual(req[1]["query_params"], test[1], "for kwargs %s" % test[0])
 
     def test_list_rows_repeated_fields(self):
@@ -6979,7 +6972,10 @@ def test_list_rows_repeated_fields(self):
         conn.api_request.assert_called_once_with(
             method="GET",
             path="/%s" % PATH,
-            query_params={"selectedFields": "color,struct"},
+            query_params={
+                "selectedFields": "color,struct",
+                "formatOptions.useInt64Timestamp": True,
+            },
             timeout=None,
         )
 
@@ -7047,7 +7043,10 @@ def test_list_rows_w_record_schema(self):
         self.assertEqual(page_token, TOKEN)
 
         conn.api_request.assert_called_once_with(
-            method="GET", path="/%s" % PATH, query_params={}, timeout=None
+            method="GET",
+            path="/%s" % PATH,
+            query_params={"formatOptions.useInt64Timestamp": True},
+            timeout=None,
         )
 
     def test_list_rows_with_missing_schema(self):
@@ -7109,7 +7108,10 @@ def test_list_rows_with_missing_schema(self):
 
             rows = list(row_iter)
             conn.api_request.assert_called_once_with(
-                method="GET", path=tabledata_path, query_params={}, timeout=None
+                method="GET",
+                path=tabledata_path,
+                query_params={"formatOptions.useInt64Timestamp": True},
+                timeout=None,
             )
             self.assertEqual(row_iter.total_rows, 3, msg=repr(table))
             self.assertEqual(rows[0].name, "Phred Phlyntstone", msg=repr(table))
diff --git a/tests/unit/test_table.py b/tests/unit/test_table.py
--- a/tests/unit/test_table.py
+++ b/tests/unit/test_table.py
@@ -2451,8 +2451,8 @@ def test_to_dataframe_timestamp_out_of_pyarrow_bounds(self):
 
         schema = [SchemaField("some_timestamp", "TIMESTAMP")]
         rows = [
-            {"f": [{"v": "81953424000.0"}]},  # 4567-01-01 00:00:00  UTC
-            {"f": [{"v": "253402214400.0"}]},  # 9999-12-31 00:00:00  UTC
+            {"f": [{"v": "81953424000000000"}]},  # 4567-01-01 00:00:00  UTC
+            {"f": [{"v": "253402214400000000"}]},  # 9999-12-31 00:00:00  UTC
         ]
         path = "/foo"
         api_request = mock.Mock(return_value={"rows": rows})
@@ -2675,9 +2675,9 @@ def test_to_dataframe_w_various_types_nullable(self):
         ]
         row_data = [
             [None, None, None, None, None, None],
-            ["1.4338368E9", "420", "1.1", u"Cash", "true", "1999-12-01"],
-            ["1.3878117E9", "2580", "17.7", u"Cash", "false", "1953-06-14"],
-            ["1.3855653E9", "2280", "4.4", u"Credit", "true", "1981-11-04"],
+            ["1433836800000000", "420", "1.1", u"Cash", "true", "1999-12-01"],
+            ["1387811700000000", "2580", "17.7", u"Cash", "false", "1953-06-14"],
+            ["1385565300000000", "2280", "4.4", u"Credit", "true", "1981-11-04"],
         ]
         rows = [{"f": [{"v": field} for field in row]} for row in row_data]
         path = "/foo"
@@ -2715,9 +2715,17 @@ def test_to_dataframe_column_dtypes(self):
             SchemaField("date", "DATE"),
         ]
         row_data = [
-            ["1.4338368E9", "420", "1.1", "1.77", u"Cash", "true", "1999-12-01"],
-            ["1.3878117E9", "2580", "17.7", "28.5", u"Cash", "false", "1953-06-14"],
-            ["1.3855653E9", "2280", "4.4", "7.1", u"Credit", "true", "1981-11-04"],
+            ["1433836800000000", "420", "1.1", "1.77", u"Cash", "true", "1999-12-01"],
+            [
+                "1387811700000000",
+                "2580",
+                "17.7",
+                "28.5",
+                u"Cash",
+                "false",
+                "1953-06-14",
+            ],
+            ["1385565300000000", "2280", "4.4", "7.1", u"Credit", "true", "1981-11-04"],
         ]
         rows = [{"f": [{"v": field} for field in row]} for row in row_data]
         path = "/foo"