Repository: googleapis/google-cloud-python

Issue:
[BigQuery] Can't insert correct TIMESTAMP.
In my environment, I can't insert correct data to the `TIMESTAMP` column.
- gcloud 0.8.0
- Python 2.7.10
- MacOS X 10.10.5

The following snippet is the smallest reproducible example.
Please let me know that it's a bug or my mistake.

``` python
import os
from datetime import datetime
from gcloud import bigquery

GCP_KEY_PATH = os.path.join(os.path.dirname(__file__), './gcpkey.json')
project_id = 'my_project'
dataset = 'tmp'
table_name = 'test'

bq = bigquery.Client.from_service_account_json(GCP_KEY_PATH, project=project_id)
table = bq.dataset(dataset).table(name=table_name)

if not table.exists():
    table.schema = [bigquery.SchemaField('time', 'TIMESTAMP')]
    table.create()

table.reload()
rows = [[datetime.now()]]
table.insert_data(rows)

```

<img width="339" alt="2015-11-28 15 47 08" src="https://cloud.githubusercontent.com/assets/899217/11450553/58c48c1a-95e7-11e5-8cf5-c4ca55a77026.png">



Discussion:
I suspect that [here](https://github.com/GoogleCloudPlatform/gcloud-python/blob/af1833476d770d553b86b18bc8bd2ab90dcf2c89/gcloud/bigquery/table.py#L657-L658) is the cause.
BigQuery's `TIMESTAMP` requires the UNIX timestamp as **seconds** but the lines convert datetime to **milliseconds**.

ref: https://cloud.google.com/bigquery/preparing-data-for-bigquery?hl=en#datatypes

Totally agree. @tseaver and I had a discussion about this before and we **thought** we had a found a decisive place indicating how the value should be sent. It seems we got it wrong.

Thank you for your comment!
Does it mean that this is a bug and you're going to fix it?
(Sorry for my poor English proficiency)

I can't say 100% it's a bug, but it seems pretty likely. It means we'll dig into verifying and then fixing if we can find a verifiable bug. (Again seems very likely.)

OK, I got it. I hope it's fixed. :smile: 

@kamatama41 The docs you link above state:

> A positive or negative decimal number. A positive number specifies the number of seconds since the epoch (1970-01-01 00:00:00 UTC), and a negative number specifies the number of seconds before the epoch. Up to 6 decimal places (microsecond precision) are preserved.

So, rather than passing milliseconds, we should just pass seconds.

FYI @tseaver I am fixing this right now

I'm also finding lots of out-of-date docs in http://gcloud-python.readthedocs.org/en/latest/bigquery-usage.html


Resolving patch:
diff --git a/gcloud/bigquery/table.py b/gcloud/bigquery/table.py
--- a/gcloud/bigquery/table.py
+++ b/gcloud/bigquery/table.py
@@ -19,6 +19,7 @@
 import six
 
 from gcloud._helpers import _datetime_from_microseconds
+from gcloud._helpers import _microseconds_from_datetime
 from gcloud._helpers import _millis_from_datetime
 from gcloud.exceptions import NotFound
 from gcloud.bigquery._helpers import _rows_from_json
@@ -657,8 +658,11 @@ def insert_data(self,
             row_info = {}
 
             for field, value in zip(self._schema, row):
-                if field.field_type == 'TIMESTAMP':
-                    value = _millis_from_datetime(value)
+                if field.field_type == 'TIMESTAMP' and value is not None:
+                    # BigQuery stores TIMESTAMP data internally as a
+                    # UNIX timestamp with microsecond precision.
+                    # Specifies the number of seconds since the epoch.
+                    value = _microseconds_from_datetime(value) * 1e-6
                 row_info[field.name] = value
 
             info = {'json': row_info}


Test patch:
diff --git a/gcloud/bigquery/test_table.py b/gcloud/bigquery/test_table.py
--- a/gcloud/bigquery/test_table.py
+++ b/gcloud/bigquery/test_table.py
@@ -1060,7 +1060,7 @@ def test_fetch_data_w_record_schema(self):
     def test_insert_data_w_bound_client(self):
         import datetime
         from gcloud._helpers import UTC
-        from gcloud._helpers import _millis_from_datetime
+        from gcloud._helpers import _microseconds_from_datetime
         from gcloud.bigquery.table import SchemaField
 
         WHEN_TS = 1437767599.006
@@ -1084,9 +1084,12 @@ def test_insert_data_w_bound_client(self):
         ]
 
         def _row_data(row):
+            joined = None
+            if row[2] is not None:
+                joined = _microseconds_from_datetime(row[2]) * 1e-6
             return {'full_name': row[0],
                     'age': row[1],
-                    'joined': _millis_from_datetime(row[2])}
+                    'joined': joined}
 
         SENT = {
             'rows': [{'json': _row_data(row)} for row in ROWS],
diff --git a/system_tests/bigquery.py b/system_tests/bigquery.py
--- a/system_tests/bigquery.py
+++ b/system_tests/bigquery.py
@@ -191,11 +191,17 @@ def test_update_table(self):
             self.assertEqual(found.mode, expected.mode)
 
     def test_load_table_then_dump_table(self):
+        import datetime
+        from gcloud._helpers import UTC
+
+        NOW_SECONDS = 1448911495.484366
+        NOW = datetime.datetime.utcfromtimestamp(
+            NOW_SECONDS).replace(tzinfo=UTC)
         ROWS = [
-            ('Phred Phlyntstone', 32),
-            ('Bharney Rhubble', 33),
-            ('Wylma Phlyntstone', 29),
-            ('Bhettye Rhubble', 27),
+            ('Phred Phlyntstone', 32, NOW),
+            ('Bharney Rhubble', 33, NOW + datetime.timedelta(seconds=10)),
+            ('Wylma Phlyntstone', 29, NOW + datetime.timedelta(seconds=20)),
+            ('Bhettye Rhubble', 27, None),
         ]
         ROW_IDS = range(len(ROWS))
         dataset = CLIENT.dataset(DATASET_NAME)
@@ -206,7 +212,8 @@ def test_load_table_then_dump_table(self):
         full_name = bigquery.SchemaField('full_name', 'STRING',
                                          mode='REQUIRED')
         age = bigquery.SchemaField('age', 'INTEGER', mode='REQUIRED')
-        table = dataset.table(TABLE_NAME, schema=[full_name, age])
+        now = bigquery.SchemaField('now', 'TIMESTAMP')
+        table = dataset.table(TABLE_NAME, schema=[full_name, age, now])
         self.assertFalse(table.exists())
         table.create()
         self.to_delete.insert(0, table)