Repository: googleapis/python-bigquery

Issue:
Client.load_table_from_dataframe() sometimes chooses invalid column type
#### Environment details

  - OS type and version: macOS 13
  - Python version: Python 3.10.7
  - pip version: pip 23.2
  - `google-cloud-bigquery` version:  3.11.4

#### Steps to reproduce

  1. Create a table with a `BIGNUMERIC` type column
  2. Use `Client.load_table_from_dataframe()` to load a column with the value `Decimal("0.12345678901234560000000000000000000000")`

#### Code example

```python
df = pd.DataFrame("TEST_COL", Decimal("0.12345678901234560000000000000000000000"))
with bigquery.Client(project=project, credentials=credentials) as client:
    load_job = client.load_table_from_dataframe(df, table_name)
    result = load_job.result()
```

#### Stack trace
```
Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "pyarrow/array.pxi", line 1044, in pyarrow.lib.Array.from_pandas
  File "pyarrow/array.pxi", line 316, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Rescaling Decimal128 value would cause data loss
```

#### Details

I poked around a bit, and I think I see the problem. To determine the column type you create an arrow table https://github.com/googleapis/python-bigquery/blob/1760e945d16163980027fecf21113cd77ddc35a1/google/cloud/bigquery/_pandas_helpers.py#L566 then lookup the type by type ID https://github.com/googleapis/python-bigquery/blob/1760e945d16163980027fecf21113cd77ddc35a1/google/cloud/bigquery/_pandas_helpers.py#L587
My test decimal value can be represented by an arrow 128-bit type
```python
>>>pyarrow.array(pd.DataFrame({"x": [Decimal('0.12345678901234560000000000000000000000')]})["x"]).type
Decimal128Type(decimal128(38, 38))
```
But AFAICT that has the same type ID as every other Decimal128Type, so it maps to "NUMERIC" in your lookup table: https://github.com/googleapis/python-bigquery/blob/1760e945d16163980027fecf21113cd77ddc35a1/google/cloud/bigquery/_pandas_helpers.py#L185
So when you convert to pyarrow before uploading, it tries to coerce using your hardcoded type for "NUMERIC" https://github.com/googleapis/python-bigquery/blob/1760e945d16163980027fecf21113cd77ddc35a1/google/cloud/bigquery/_pandas_helpers.py#L380
and fails because it needed `decimal128(38, 38)` to represent the value, but is given  `decimal128(38, 9)`.  I think you will need to detect if a 128 bit type can't fit in your NUMERIC subtype and upgrade those to BIGNUMERIC.


Hope that helps!


Discussion:
Thanks for submitting this and looking at it so thoroughly.

Resolving patch:
diff --git a/google/cloud/bigquery/_pandas_helpers.py b/google/cloud/bigquery/_pandas_helpers.py
--- a/google/cloud/bigquery/_pandas_helpers.py
+++ b/google/cloud/bigquery/_pandas_helpers.py
@@ -23,6 +23,7 @@
 import warnings
 from typing import Any, Union
 
+
 from google.cloud.bigquery import _pyarrow_helpers
 from google.cloud.bigquery import _versions_helpers
 from google.cloud.bigquery import schema
@@ -485,7 +486,6 @@ def augment_schema(dataframe, current_bq_schema):
     # pytype: disable=attribute-error
     augmented_schema = []
     unknown_type_fields = []
-
     for field in current_bq_schema:
         if field.field_type is not None:
             augmented_schema.append(field)
@@ -515,6 +515,8 @@ def augment_schema(dataframe, current_bq_schema):
         else:
             detected_mode = field.mode
             detected_type = _pyarrow_helpers.arrow_scalar_ids_to_bq(arrow_table.type.id)
+            if detected_type == "NUMERIC" and arrow_table.type.scale > 9:
+                detected_type = "BIGNUMERIC"
 
         if detected_type is None:
             unknown_type_fields.append(field)


Test patch:
diff --git a/tests/unit/test_client.py b/tests/unit/test_client.py
--- a/tests/unit/test_client.py
+++ b/tests/unit/test_client.py
@@ -8891,6 +8891,49 @@ def test_load_table_from_dataframe_with_csv_source_format(self):
         sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
         assert sent_config.source_format == job.SourceFormat.CSV
 
+    @unittest.skipIf(pandas is None, "Requires `pandas`")
+    @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+    def test_load_table_from_dataframe_w_higher_scale_decimal128_datatype(self):
+        from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+        from google.cloud.bigquery import job
+        from google.cloud.bigquery.schema import SchemaField
+        from decimal import Decimal
+
+        client = self._make_client()
+        dataframe = pandas.DataFrame({"x": [Decimal("0.1234567891")]})
+        load_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+        )
+
+        get_table_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.get_table", autospec=True
+        )
+        with load_patch as load_table_from_file, get_table_patch:
+            client.load_table_from_dataframe(
+                dataframe, self.TABLE_REF, location=self.LOCATION
+            )
+
+        load_table_from_file.assert_called_once_with(
+            client,
+            mock.ANY,
+            self.TABLE_REF,
+            num_retries=_DEFAULT_NUM_RETRIES,
+            rewind=True,
+            size=mock.ANY,
+            job_id=mock.ANY,
+            job_id_prefix=None,
+            location=self.LOCATION,
+            project=None,
+            job_config=mock.ANY,
+            timeout=DEFAULT_TIMEOUT,
+        )
+
+        sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+        assert sent_config.source_format == job.SourceFormat.PARQUET
+        assert tuple(sent_config.schema) == (
+            SchemaField("x", "BIGNUMERIC", "NULLABLE", None),
+        )
+
     def test_load_table_from_json_basic_use(self):
         from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
         from google.cloud.bigquery import job