Repository: googleapis/google-cloud-python

Issue:
BigQuery Time Values May Be Incorrect
On the [`_datetime_from_json` function](https://github.com/GoogleCloudPlatform/gcloud-python/blob/62fd8310259bc9761a6eb3ceb0d2db9677814f6a/gcloud/bigquery/table.py#L764), @arterrey [commented](https://github.com/GoogleCloudPlatform/gcloud-python/commit/41e64395661a31af8352fb2358e5c95ff5437830#commitcomment-13104217) that

> I'm getting values out of BigQuery in seconds. So \* 1000.0 is not enough - it needs to be \* 1000000.0 to work. However, I'm not sure if that is a bug in the BigQuery API or a bug in this client library.

This contradicts the note in the code that

> Field value will be in milliseconds.



Discussion:
@arterrey Can you elaborate where you are "getting values out of BigQuery in seconds"? Like an example use of this library and how the output differs from what you expect?

The field value from the web interface is "2012-06-12 08:05:14 UTC". From the pdb session below you can see that the value returns a dattime in 1970 rather then 2012.

```
(Pdb) list 762
757     def _bool_from_json(value, field):
758         if _not_null(value, field):
759             return value.lower() in ['t', 'true', '1']
760     
761     
762     def _datetime_from_json(value, field):
763         if _not_null(value, field):
764             import pdb; pdb.set_trace()
765             # Field value will be in milliseconds.        
766  ->         return _datetime_from_microseconds(1000.0 * float(value))
767     
(Pdb) value
u'1.33948831400937E9'
(Pdb) float(value)
1339488314.00937
(Pdb) _datetime_from_microseconds(1000.0 * float(value))
datetime.datetime(1970, 1, 16, 12, 4, 48, 314009, tzinfo=<UTC>)
(Pdb) _datetime_from_microseconds(1000000.0 * float(value))
datetime.datetime(2012, 6, 12, 8, 5, 14, 9370, tzinfo=<UTC>)
(Pdb) 
```

That's really great info! Thanks for showing the `pdb` stack.

It looks like

``` python
value == 1339.488314009370
```

Which request did you make to retrieve this data? Can you provide a code snippet?

I think you mean

``` python
    value == 1339488314.00937
```

just using `fetch_data`...

``` python
    def fetch_table(data_set, table_name):

        table = data_set.table(table_name)
        assert table.exists()
        table.reload()

        next_page = None

        while (True):
            rows, total_count, next_page = table.fetch_data(page_token=next_page)
            for row in rows:

                # use schema to convert tuple into dict
                row_dict = {}
                for i in range(len(table.schema)):
                    field = table.schema[i]
                    row_dict[field.name] = row[i]

                yield PathableDict(row_dict)

            if next_page is None:
                break
```

Sorry I overlooked the actual value:

``` python
(Pdb) value
u'1.33948831400937E9'
```

It seems that instead of

``` python
datetime.timedelta(microseconds=1000.0 * float(value))
```

we should just be using

``` python
datetime.timedelta(seconds=float(value))
```

Like you said, the value is seconds, not milliseconds.

I couldn't find any evidence of "seconds" in the discovery doc:
https://www.googleapis.com/discovery/v1/apis/bigquery/v2/rest

FWIW the `bigquery.tabledata.list` returns a `TableDataList`, and the schema for that type has `TableDataList.rows` a list of `TableRow`, which in turn has `TableRow.f` a list of `TableCell` which then is an `any` type (which is kind of a black hole).

I'm off to bed for now but will try to throw in some test data (from our system tests) and inspect the output on the APIs explorer:
https://developers.google.com/apis-explorer/#p/bigquery/v2/bigquery.tabledata.list

Also FWIW our link in our docstring:
https://cloud.google.com/bigquery/reference/rest/v2/tabledata/list
is broken. It should be:
https://cloud.google.com/bigquery/docs/reference/v2/tabledata/list

---

Really just

``` python
rows, total_count, next_page = table.fetch_data(page_token=next_page)
```

was what I was looking for. Thanks!

I just created a table with the following schema:

``` json
[
   {
    "name": "Timestamp",
    "type": "TIMESTAMP",
    "mode": "REQUIRED"
   },
   {
    "name": "Description",
    "type": "STRING",
    "mode": "REQUIRED"
   },
   {
    "name": "Quantity",
    "type": "INTEGER",
    "mode": "REQUIRED"
   },
   {
    "name": "Unit_Price",
    "type": "FLOAT",
    "mode": "REQUIRED"
   }
]
```

and uploaded the following data:

``` bash
$ cat /tmp/with_timestamp.csv 
2015-09-09T17:59:22Z,Testing,1,2.5
```

In the developer console, the data looks like:

``` text
Row  Timestamp                Description  Quantity Unit_Price   
1    2015-09-09 17:59:22 UTC  Testing      1        2.5
```

In the API explorer, `tabledata.list` returns:

``` json
{
 "kind": "bigquery#tableDataList",
 "etag": "\"m-9bZ2XqlR7n8jGvqgVmLOTsm1s/xddlFZR3Ob1xPfHtKV-u3wV2TLU\"",
 "totalRows": "1",
 "rows": [
  {
   "f": [
    {
     "v": "1.441821562E9"
    },
    {
     "v": "Testing"
    },
    {
     "v": "1"
    },
    {
     "v": "2.5"
    }
   ]
  }
 ]
}
```

The [Preparing data for bigquery docs](https://cloud.google.com/bigquery/preparing-data-for-bigquery#datatypes) don't specify how timestamps are _returned_, only the formats which can be used to load the data.  It does say,

> BigQuery stores TIMESTAMP data internally as a UNIX timestamp with microsecond precision.

So what shall we do? Also, how does `2015-09-09T17:59:22Z,Testing,1,2.5` make it through our pipeline?

Given that the values are always returned as scientific-formatted floats of seconds (unlike `datastore` timestamps, which are stored as microseconds), we should be using `float(value)`.

> How does `2015-09-09T17:59:22Z,Testing,1,2.5` make it through our pipeline?

ISO-8660 strings are one of the [supported formats](https://cloud.google.com/bigquery/preparing-data-for-bigquery#datatypes) for loading `TIMESTAMP` data.

I mean what is the codepath.

`Client.load_table_from_storage` could trigger that path.


Resolving patch:
diff --git a/gcloud/bigquery/table.py b/gcloud/bigquery/table.py
--- a/gcloud/bigquery/table.py
+++ b/gcloud/bigquery/table.py
@@ -411,7 +411,7 @@ def create(self, client=None):
         """API call:  create the dataset via a PUT request
 
         See:
-        https://cloud.google.com/bigquery/reference/rest/v2/tables/insert
+        https://cloud.google.com/bigquery/docs/reference/v2/tables/insert
 
         :type client: :class:`gcloud.bigquery.client.Client` or ``NoneType``
         :param client: the client to use.  If not passed, falls back to the
@@ -552,7 +552,7 @@ def delete(self, client=None):
         """API call:  delete the table via a DELETE request
 
         See:
-        https://cloud.google.com/bigquery/reference/rest/v2/tables/delete
+        https://cloud.google.com/bigquery/docs/reference/v2/tables/delete
 
         :type client: :class:`gcloud.bigquery.client.Client` or ``NoneType``
         :param client: the client to use.  If not passed, falls back to the
@@ -565,7 +565,7 @@ def fetch_data(self, max_results=None, page_token=None, client=None):
         """API call:  fetch the table data via a GET request
 
         See:
-        https://cloud.google.com/bigquery/reference/rest/v2/tabledata/list
+        https://cloud.google.com/bigquery/docs/reference/v2/tabledata/list
 
         .. note::
 
@@ -631,7 +631,7 @@ def insert_data(self,
         """API call:  insert table data via a POST request
 
         See:
-        https://cloud.google.com/bigquery/reference/rest/v2/tabledata/insertAll
+        https://cloud.google.com/bigquery/docs/reference/v2/tabledata/insertAll
 
         :type rows: list of tuples
         :param rows: row data to be inserted
@@ -761,8 +761,8 @@ def _bool_from_json(value, field):
 
 def _datetime_from_json(value, field):
     if _not_null(value, field):
-        # Field value will be in milliseconds.
-        return _datetime_from_microseconds(1000.0 * float(value))
+        # value will be a float in seconds, to microsecond precision, in UTC.
+        return _datetime_from_microseconds(1e6 * float(value))
 
 
 def _record_from_json(value, field):


Test patch:
diff --git a/gcloud/bigquery/test_table.py b/gcloud/bigquery/test_table.py
--- a/gcloud/bigquery/test_table.py
+++ b/gcloud/bigquery/test_table.py
@@ -824,7 +824,6 @@ def test_fetch_data_w_bound_client(self):
         import datetime
         from gcloud._helpers import UTC
         from gcloud.bigquery.table import SchemaField
-        from gcloud._helpers import _millis_from_datetime
 
         PATH = 'projects/%s/datasets/%s/tables/%s/data' % (
             self.PROJECT, self.DS_NAME, self.TABLE_NAME)
@@ -835,6 +834,11 @@ def test_fetch_data_w_bound_client(self):
         WHEN_2 = WHEN + datetime.timedelta(seconds=2)
         ROWS = 1234
         TOKEN = 'TOKEN'
+
+        def _bigquery_timestamp_float_repr(ts_float):
+            # Preserve microsecond precision for E+09 timestamps
+            return '%0.15E' % (ts_float,)
+
         DATA = {
             'totalRows': ROWS,
             'pageToken': TOKEN,
@@ -842,17 +846,17 @@ def test_fetch_data_w_bound_client(self):
                 {'f': [
                     {'v': 'Phred Phlyntstone'},
                     {'v': '32'},
-                    {'v': _millis_from_datetime(WHEN)},
+                    {'v': _bigquery_timestamp_float_repr(WHEN_TS)},
                 ]},
                 {'f': [
                     {'v': 'Bharney Rhubble'},
                     {'v': '33'},
-                    {'v': _millis_from_datetime(WHEN_1)},
+                    {'v': _bigquery_timestamp_float_repr(WHEN_TS + 1)},
                 ]},
                 {'f': [
                     {'v': 'Wylma Phlyntstone'},
                     {'v': '29'},
-                    {'v': _millis_from_datetime(WHEN_2)},
+                    {'v': _bigquery_timestamp_float_repr(WHEN_TS + 2)},
                 ]},
                 {'f': [
                     {'v': 'Bhettye Rhubble'},
@@ -861,6 +865,7 @@ def test_fetch_data_w_bound_client(self):
                 ]},
             ]
         }
+
         conn = _Connection(DATA)
         client = _Client(project=self.PROJECT, connection=conn)
         dataset = _Dataset(client)