Repository: googleapis/python-bigquery

Issue:
BigQuery: to_dataframe does not respect date fields
#### Environment details

1. google-cloud-bigquery: version 1.22
2. OS type and version: Centos 7
3. Python version: 3.6.9

#### Steps to reproduce

  1. Download a table from BigQuery to a pandas Dataframe

#### Code example

```python
df = bq_client.list_rows(table).to_dataframe()
df.types

contractId                       int64
contractTypeId                   int64
affiliateId                      int64
invoicePeriodId                  int64
startDate                       **object**
endDate                         **object**
contractName                    object
details                         object
updated            datetime64[ns, UTC]
created            datetime64[ns, UTC]
dtype: object
```
Timestamps seem to work fine, but **date** is just being treated as a string.

Table schema is defined as:
```python
'contract' : [
            bigquery.SchemaField("contractId", "INTEGER", mode="REQUIRED"),
            bigquery.SchemaField("contractTypeId", "INTEGER", mode="REQUIRED"),
            bigquery.SchemaField("affiliateId", "INTEGER", mode="REQUIRED"),
            bigquery.SchemaField("invoicePeriodId", "INTEGER", mode="REQUIRED"),
            bigquery.SchemaField("startDate", "DATE", mode="REQUIRED"),
            bigquery.SchemaField("endDate", "DATE", mode="NULLABLE"),
            bigquery.SchemaField("contractName", "STRING", mode="NULLABLE"),
            bigquery.SchemaField("details", "STRING", mode="NULLABLE"),
            bigquery.SchemaField("created", "TIMESTAMP", mode="REQUIRED"),
            bigquery.SchemaField("updated", "TIMESTAMP", mode="REQUIRED"),
        ]
```
This should have anything to do with pagination, because the _startDate_ field is required; also the table only has 20 records.


Discussion:
I'm also seeing problem using _load_table_from_dataframe_ with datetime columns.  If I specificy the schema as part of the LoadJobConfig (specifying the column as **DATE**, it works fine.

However if I fail to specify the schema, it identifies the column as **DATETIME** and creates invalid dates:
```
Row	| startDate	
1	| 0021-04-25T22:13:48.360448
2	| 0021-04-25T22:13:48.360448
3	| 0021-08-13T11:42:29.163840
```

As opposed to :
```
Row	| startDate	
1	| 2017-07-01
2	| 2017-07-01
3	| 2018-01-01
```
Pandas dtype is datetime64[ns]
@QuinRiva  I have tried to reproduce the issue related to `load_table_from_dataframe` but unable to do that, here is my sample code and dataframe:

    contractId | contractTypeId |  startDate    |   endDate   |   created
        2      |    2           |  2019-12-06   | 2019-12-06  | 2019-12-06 17:09:11.317689+00:00
        1      |    1           |  2019-12-06   | 2019-12-06  | 2019-12-06 17:09:11.317689+00:00

df.types

    contractId                      int64
    contractTypeId                  int64
    startDate                      object
    endDate                        object
    created           datetime64[ns, UTC]

Sample code:

    from google.cloud import bigquery
    client = bigquery.Client()
    destination_table = bigquery.table('your-project.your_dataset.your_table')
    client.load_table_from_dataframe (df,destination_table)

It creates table with the valid dates and datetime.
If it doesn't help than could you please share reproducible code?


I've been having the same issue with `load_table_from_dataframe` since updating dependencies. I have uncovered the real problem. Some of my recent timestamps throw errors like this one:

`Cannot return an invalid datetime value of 1575982560000000 microseconds relative to the Unix epoch`

What's actually going on here is that I have a pandas DataFrame with values like this:
 
`'2019-12-10 12:56:00', dtype='datetime64[ns]`

which, in ns, is `1575982560000000000`.

In other words, somewhere between being a dataframe and ending up in BigQuery, the underlying value stored for this date is being interpreted as microseconds since the Unix epoch rather than _nanoseconds_!

I recently had an issue with the same code throwing errors after a pyarrow update, which might be related - I ended up fixing that one by rounding my timestamps to ms-precision before uploading to BigQuery (see https://stackoverflow.com/questions/53333742/schema-conflict-when-storing-dataframes-with-datetime-objects-using-load-table-f for an example).

EDIT: it appears that something has changed in terms of the BigQuery type being inferred from pandas. Previously, the type used was `TIMESTAMP`. Now, it appears to be `DATETIME`. I am trying to fix my specific issue by using a partial schema - in case it helps anyone else:

```
        job_config = LoadJobConfig()

        dt_columns = df.columns[df.dtypes == "datetime64[ns]"]

        job_config.schema = [SchemaField(col, "TIMESTAMP") for col in dt_columns]
``` 
@tswast , @plamut  please comment, is it related to pyarrow?
Actually, let's specify a second issue for `load_table_from_dataframe`. https://github.com/googleapis/google-cloud-python/issues/9996
@QuinRiva is the behavior of `to_dataframe` any different when using the BigQuery Storage API?

You might also want to try setting the `dtypes` argument to `to_dataframe`. We don't have any logic to convert from the API response types to pandas `dtypes` unless the `dtypes` argument is specified.
@tswast I expect the current behavior with the BigQuery Storage API that the dtype will still be "object" but at least with Arrow (and I assume Avro) the actual objects will python datetime.date object instead of strings.

@QuinRiva could you specify which dtype/object type you are expecting for the date columns?  

@emkornfield Another issue has filed by user https://github.com/googleapis/python-bigquery/issues/136 which is duplicate of this issue.

As per comment received from user on the [issue](https://github.com/googleapis/python-bigquery/issues/136#issuecomment-644141436),  Manually setting the dtype argument works but user wants that automated.
@HemangChothani I think this might be fixed after https://github.com/googleapis/python-bigquery/commit/e75ff8297c65981545b097f75a17cf9e78ac6772 (is this in the latest release?).  Otherwise I think it is possible this might be going through a different code path and not using pyarrow through the storage API (and not using arrow data so it still might be maintaining the string).  pyarrow will covert date types to `datetime.date` objects without intervention when using `pa.record_batch.to_dataframe()`
@emkornfield I have tried with the master branch as feature has not released, but still `to_dataframe` is the culprit. 

pyarrow works fine as `to_arrow` returns `date32` datatype`[pyarrow.RecordBatch dt: date32[day]]`
but line given below convert to df and return  df.dtype ['column'] = object https://github.com/googleapis/python-bigquery/blob/de9998fe85eea5e69be39cfe0075d3097ce9ec4a/google/cloud/bigquery/table.py#L1725
@HemangChothani I expect it to be an object (but the object should be datetime.date).  If conversion to dtype=datetime64  is desired behavior I think you need to call [`record_batch.to_pandas(date_as_object=False)`](https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatch.html#pyarrow.RecordBatch.to_pandas)
@emkornfield I have completely ignored that and expect it to be a `datetime64`, thanks for the clarification, is there any plan to expose this parameter for users, if not than can we close this issue as it's resolved? 
> is there any plan to expose this parameter for users, if not than can we close this issue as it's resolved?
The parameter is public in the pyarrow as far as I can tell.  It seems like at least some users of BQ expect date to become datetime64 I would therefor think it is a good idea to expose this as a parameter for BQ, but I don't know the exact policies that are followed.

Resolving patch:
diff --git a/google/cloud/bigquery/job.py b/google/cloud/bigquery/job.py
--- a/google/cloud/bigquery/job.py
+++ b/google/cloud/bigquery/job.py
@@ -3320,6 +3320,7 @@ def to_dataframe(
         dtypes=None,
         progress_bar_type=None,
         create_bqstorage_client=True,
+        date_as_object=True,
     ):
         """Return a pandas DataFrame from a QueryJob
 
@@ -3350,9 +3351,9 @@ def to_dataframe(
                 for details.
 
                 ..versionadded:: 1.11.0
-            create_bqstorage_client (bool):
-                Optional. If ``True`` (default), create a BigQuery Storage API
-                client using the default API settings. The BigQuery Storage API
+            create_bqstorage_client (Optional[bool]):
+                If ``True`` (default), create a BigQuery Storage API client
+                using the default API settings. The BigQuery Storage API
                 is a faster way to fetch rows from BigQuery. See the
                 ``bqstorage_client`` parameter for more information.
 
@@ -3360,6 +3361,12 @@ def to_dataframe(
 
                 ..versionadded:: 1.24.0
 
+            date_as_object (Optional[bool]):
+                If ``True`` (default), cast dates to objects. If ``False``, convert
+                to datetime64[ns] dtype.
+
+                ..versionadded:: 1.26.0
+
         Returns:
             A :class:`~pandas.DataFrame` populated with row data and column
             headers from the query results. The column headers are derived
@@ -3373,6 +3380,7 @@ def to_dataframe(
             dtypes=dtypes,
             progress_bar_type=progress_bar_type,
             create_bqstorage_client=create_bqstorage_client,
+            date_as_object=date_as_object,
         )
 
     def __iter__(self):
diff --git a/google/cloud/bigquery/table.py b/google/cloud/bigquery/table.py
--- a/google/cloud/bigquery/table.py
+++ b/google/cloud/bigquery/table.py
@@ -1633,6 +1633,7 @@ def to_dataframe(
         dtypes=None,
         progress_bar_type=None,
         create_bqstorage_client=True,
+        date_as_object=True,
     ):
         """Create a pandas DataFrame by loading all pages of a query.
 
@@ -1673,9 +1674,9 @@ def to_dataframe(
                   progress bar as a graphical dialog box.
 
                 ..versionadded:: 1.11.0
-            create_bqstorage_client (bool):
-                Optional. If ``True`` (default), create a BigQuery Storage API
-                client using the default API settings. The BigQuery Storage API
+            create_bqstorage_client (Optional[bool]):
+                If ``True`` (default), create a BigQuery Storage API client
+                using the default API settings. The BigQuery Storage API
                 is a faster way to fetch rows from BigQuery. See the
                 ``bqstorage_client`` parameter for more information.
 
@@ -1683,6 +1684,12 @@ def to_dataframe(
 
                 ..versionadded:: 1.24.0
 
+            date_as_object (Optional[bool]):
+                If ``True`` (default), cast dates to objects. If ``False``, convert
+                to datetime64[ns] dtype.
+
+                ..versionadded:: 1.26.0
+
         Returns:
             pandas.DataFrame:
                 A :class:`~pandas.DataFrame` populated with row data and column
@@ -1722,7 +1729,7 @@ def to_dataframe(
                 bqstorage_client=bqstorage_client,
                 create_bqstorage_client=create_bqstorage_client,
             )
-            df = record_batch.to_pandas()
+            df = record_batch.to_pandas(date_as_object=date_as_object)
             for column in dtypes:
                 df[column] = pandas.Series(df[column], dtype=dtypes[column])
             return df
@@ -1799,6 +1806,7 @@ def to_dataframe(
         dtypes=None,
         progress_bar_type=None,
         create_bqstorage_client=True,
+        date_as_object=True,
     ):
         """Create an empty dataframe.
 
@@ -1807,6 +1815,7 @@ def to_dataframe(
             dtypes (Any): Ignored. Added for compatibility with RowIterator.
             progress_bar_type (Any): Ignored. Added for compatibility with RowIterator.
             create_bqstorage_client (bool): Ignored. Added for compatibility with RowIterator.
+            date_as_object (bool): Ignored. Added for compatibility with RowIterator.
 
         Returns:
             pandas.DataFrame: An empty :class:`~pandas.DataFrame`.


Test patch:
diff --git a/tests/unit/test_job.py b/tests/unit/test_job.py
--- a/tests/unit/test_job.py
+++ b/tests/unit/test_job.py
@@ -5504,7 +5504,15 @@ def test_to_dataframe_column_dtypes(self):
             },
         }
         row_data = [
-            ["1.4338368E9", "420", "1.1", "1.77", "Cash", "true", "1999-12-01"],
+            [
+                "1.4338368E9",
+                "420",
+                "1.1",
+                "1.77",
+                "Cto_dataframeash",
+                "true",
+                "1999-12-01",
+            ],
             ["1.3878117E9", "2580", "17.7", "28.5", "Cash", "false", "1953-06-14"],
             ["1.3855653E9", "2280", "4.4", "7.1", "Credit", "true", "1981-11-04"],
         ]
@@ -5533,6 +5541,69 @@ def test_to_dataframe_column_dtypes(self):
         self.assertEqual(df.complete.dtype.name, "bool")
         self.assertEqual(df.date.dtype.name, "object")
 
+    @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+    @unittest.skipIf(pandas is None, "Requires `pandas`")
+    def test_to_dataframe_column_date_dtypes(self):
+        begun_resource = self._make_resource()
+        query_resource = {
+            "jobComplete": True,
+            "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+            "totalRows": "1",
+            "schema": {"fields": [{"name": "date", "type": "DATE"}]},
+        }
+        row_data = [
+            ["1999-12-01"],
+        ]
+        rows = [{"f": [{"v": field} for field in row]} for row in row_data]
+        query_resource["rows"] = rows
+        done_resource = copy.deepcopy(begun_resource)
+        done_resource["status"] = {"state": "DONE"}
+        connection = _make_connection(
+            begun_resource, query_resource, done_resource, query_resource
+        )
+        client = _make_client(project=self.PROJECT, connection=connection)
+        job = self._make_one(self.JOB_ID, self.QUERY, client)
+        df = job.to_dataframe(date_as_object=False, create_bqstorage_client=False)
+
+        self.assertIsInstance(df, pandas.DataFrame)
+        self.assertEqual(len(df), 1)  # verify the number of rows
+        exp_columns = [field["name"] for field in query_resource["schema"]["fields"]]
+        self.assertEqual(list(df), exp_columns)  # verify the column names
+
+        self.assertEqual(df.date.dtype.name, "datetime64[ns]")
+
+    @unittest.skipIf(pandas is None, "Requires `pandas`")
+    def test_to_dataframe_column_date_dtypes_wo_pyarrow(self):
+        begun_resource = self._make_resource()
+        query_resource = {
+            "jobComplete": True,
+            "jobReference": {"projectId": self.PROJECT, "jobId": self.JOB_ID},
+            "totalRows": "1",
+            "schema": {"fields": [{"name": "date", "type": "DATE"}]},
+        }
+        row_data = [
+            ["1999-12-01"],
+        ]
+        rows = [{"f": [{"v": field} for field in row]} for row in row_data]
+        query_resource["rows"] = rows
+        done_resource = copy.deepcopy(begun_resource)
+        done_resource["status"] = {"state": "DONE"}
+        connection = _make_connection(
+            begun_resource, query_resource, done_resource, query_resource
+        )
+        client = _make_client(project=self.PROJECT, connection=connection)
+        job = self._make_one(self.JOB_ID, self.QUERY, client)
+
+        with mock.patch("google.cloud.bigquery.table.pyarrow", None):
+            df = job.to_dataframe(date_as_object=False, create_bqstorage_client=False)
+
+        self.assertIsInstance(df, pandas.DataFrame)
+        self.assertEqual(len(df), 1)  # verify the number of rows
+        exp_columns = [field["name"] for field in query_resource["schema"]["fields"]]
+        self.assertEqual(list(df), exp_columns)  # verify the column names
+
+        self.assertEqual(df.date.dtype.name, "object")
+
     @unittest.skipIf(pandas is None, "Requires `pandas`")
     @unittest.skipIf(tqdm is None, "Requires `tqdm`")
     @mock.patch("tqdm.tqdm")