Repository: googleapis/google-cloud-python

Issue:
BigQuery: Deprecate automatic schema conversion in `load_table_from_dataframe`
After discussion around https://github.com/googleapis/google-cloud-python/issues/9024, I'm coming to realize that there are a lot of inconsistencies with the pandas DataFrame serialization when we have to autodetect the schema. I propose that we warn when we are given a DataFrame but can't determine the correct schema.

I realize this will be a step backwards in terms of usability. I think the following feature requests are needed to be prioritized if we proceed with this deprecation:

- [x] https://github.com/googleapis/google-cloud-python/issues/8140 Allow partial schemas, so that someone can just override certain `object` dtype columns, for example.
- [x] https://github.com/googleapis/google-cloud-python/issues/8142 Get table schema if not supplied. This covers the case when appending rows to an existing table.
- [x] https://github.com/googleapis/google-cloud-python/issues/9044 Generate a (partial) schema for known dtypes, and merge with the provided schema.
- [x] https://github.com/googleapis/google-cloud-python/issues/5572#issuecomment-524012299 Add index(es) to serialized file if specified in `job_config.schema`.


Discussion:
#9049 is pending (needs unit test updates), which will make me slightly more comfortable with this deprecation. My current thought for the "load dataframe" algorithm is:

* If schema is not specified, get table schema #8142
* Try to automatically determine the types for any remaining columns. #9044 
* Did we succeed in detecting the full schema? If so, great. Continue with upload.
* Oops, we weren't able to determine the schema. Fallback to `to_parquet` logic and show deprecation warning (this issue).
Added https://github.com/googleapis/google-cloud-python/issues/5572#issuecomment-524012299 as another blocker. If we don't provide a way to explicitly serializing index(es), we actually lose the ability to write indexes outside the deprecated path.
Thought: Do we want to 100% deprecate the `to_parquet` code path? What if someone explicitly chooses to use an autodetected schema? Maybe if autodetect is set, we don't warn if we weren't able to determine a full schema? That'd certainly be more explicit that "hey we're gonna try to detect some data types based on the actual values, but we might not get it exactly how you want" if we warned unless autodetect was true in that case.
Hmm ... I would still find it useful to be warned about any potential problems with schema detection, even I explicitly instruct the method to detect it for me. Suppressing the warning might make me think that everything went well behind the scenes.

How about having an `ignore_schema_warnings` (default: `False`) option to explicitly ignore any warnings? With an explanation that if an explicit schema is provided (and thus the autodetect is not used), the option has no effect?
> Suppressing the warning might make me think that everything went well behind the scenes.

In a lot of cases, `to_parquet` works fine. It's just that we've been hit with a bunch of issues recently around missing struct support. I do wish we had a way to determine when that case might be hit without inspecting the actual values (slow!).

My thought with https://github.com/googleapis/google-cloud-python/issues/9042#issuecomment-524162283 is that there are still going to be times when you have an arbitrary DataFrame and can't exactly guess the schema and want to explicitly opt-in to whatever pandas is doing to convert DataFrames to Parquet files, even if we know it might not always pick the type we want (such as confusing nullable integer columns and float columns).
I am not that familiar with what a "typical" use case is, but the scenario described above sounds plausible. And we can still mention the potential misdetection of schema in the "opt-in"' parameter's docstring, and that warnings will not be issued. Users will then know that they are explicitly turning off this aspect.

Resolving patch:
diff --git a/bigquery/google/cloud/bigquery/client.py b/bigquery/google/cloud/bigquery/client.py
--- a/bigquery/google/cloud/bigquery/client.py
+++ b/bigquery/google/cloud/bigquery/client.py
@@ -1572,6 +1572,16 @@ def load_table_from_dataframe(
             dataframe, job_config.schema
         )
 
+        if not job_config.schema:
+            # the schema could not be fully detected
+            warnings.warn(
+                "Schema could not be detected for all columns. Loading from a "
+                "dataframe without a schema will be deprecated in the future, "
+                "please provide a schema.",
+                PendingDeprecationWarning,
+                stacklevel=2,
+            )
+
         tmpfd, tmppath = tempfile.mkstemp(suffix="_job_{}.parquet".format(job_id[:8]))
         os.close(tmpfd)
 


Test patch:
diff --git a/bigquery/tests/unit/test_client.py b/bigquery/tests/unit/test_client.py
--- a/bigquery/tests/unit/test_client.py
+++ b/bigquery/tests/unit/test_client.py
@@ -5579,6 +5579,41 @@ def test_load_table_from_dataframe_unknown_table(self):
             job_config=mock.ANY,
         )
 
+    @unittest.skipIf(pandas is None, "Requires `pandas`")
+    @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+    def test_load_table_from_dataframe_no_schema_warning(self):
+        client = self._make_client()
+
+        # Pick at least one column type that translates to Pandas dtype
+        # "object". A string column matches that.
+        records = [{"name": "Monty", "age": 100}, {"name": "Python", "age": 60}]
+        dataframe = pandas.DataFrame(records)
+
+        get_table_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.get_table",
+            autospec=True,
+            side_effect=google.api_core.exceptions.NotFound("Table not found"),
+        )
+        load_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+        )
+        pyarrow_patch = mock.patch("google.cloud.bigquery.client.pyarrow", None)
+        catch_warnings = warnings.catch_warnings(record=True)
+
+        with get_table_patch, load_patch, pyarrow_patch, catch_warnings as warned:
+            client.load_table_from_dataframe(
+                dataframe, self.TABLE_REF, location=self.LOCATION
+            )
+
+        matches = [
+            warning
+            for warning in warned
+            if warning.category in (DeprecationWarning, PendingDeprecationWarning)
+            and "could not be detected" in str(warning)
+            and "please provide a schema" in str(warning)
+        ]
+        assert matches, "A missing schema deprecation warning was not raised."
+
     @unittest.skipIf(pandas is None, "Requires `pandas`")
     @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
     def test_load_table_from_dataframe_struct_fields_error(self):