Repository: googleapis/python-bigquery

Issue:
load_table_from_dataframe does not error out when `nan` in a required column - Million dollar bug
When loading a pandas dataframe into Bigquery that contains a nan in a required column, the upload succeeds but the resulting table is not representative of the dataframe. The values in the column containing the `nan` are unordered and the user is not aware of it.

#### Environment details

  - OS type and version:
  - Python version: Python 3.11.4
  - pip version: pip 23.2.1
  - `google-cloud-bigquery` version: 
  - Name: google-cloud-bigquery
  ```
Version: 3.12.0
Summary: Google BigQuery API client library
Home-page: https://github.com/googleapis/python-bigquery
Author: Google LLC
Author-email: googleapis-packages@google.com
License: Apache 2.0
Requires: google-api-core, google-cloud-core, google-resumable-media, grpcio, grpcio, packaging, proto-plus, protobuf, python-dateutil, requests
Required-by: google-cloud-aiplatform, pandas-gbq
  ```

#### Steps to reproduce

  1. Run the code below

#### Code example

```python
from google.cloud import bigquery 
import pandas as pd
import numpy as np

df = pd.DataFrame([["hello", "string"], ["hello2", np.nan], ["hello3", "valid"], ["hello4", "valid2"]], columns=["image_uri", "phash"])

client = bigquery.Client(project="project")
job_config = bigquery.LoadJobConfig(
    schema=[
        bigquery.SchemaField("image_uri", "STRING", mode="REQUIRED"),
        bigquery.SchemaField("phash", "STRING", mode="REQUIRED"),
    ]
)
job = client.load_table_from_dataframe(
    df, "foo.foo_bar", job_config=job_config
)
job.result()

df_read = pd.read_gbq("foo.foo_bar", project_id="project")
```
![image](https://github.com/googleapis/python-bigquery/assets/29891099/3540b6d3-879b-4720-ba3a-f0c06772b660)
![image](https://github.com/googleapis/python-bigquery/assets/29891099/43c2d2c4-1f63-4128-8c2c-b48fd3f7ac04)




Discussion:
seems like there is an issue with pyarrow library.  Following code show the reproduction of the same bug. 
```python
    import pyarrow as pa
    import pyarrow.parquet
    import pandas as pd

    fields = [pa.field('id', pa.string(), nullable=False),
              pa.field('name', pa.string(), nullable=False)]
    array = [pa.array(['10', '11', '12', '13']),
             pa.array(['AAA', None, 'BBB', 'CCC'])]
    table = pa.Table.from_arrays(array, schema=pa.schema(fields))
    pyarrow.parquet.write_table(table, 'test_arrow.parquet', compression='SNAPPY', use_compliant_nested_type=True)
    df = pd.read_parquet("/Users/fki/Documents/git/Demo/bq_api/test_arrow.parquet", engine='pyarrow')
    print("\n\n\n")
    print(df)
```

## Possible solution

 until issue is fixed in Pyarrow.  Disable the nullability for schema in pyarrow table.  This will throw unhandled exception though. 

**_pandas_helper.py**

```python
def bq_to_arrow_field(bq_field, array_type=None):
    """Return the Arrow field, corresponding to a given BigQuery column.

    Returns:
        None: if the Arrow type cannot be determined.
    """
    arrow_type = bq_to_arrow_data_type(bq_field)
    if arrow_type is not None:
        if array_type is not None:
            arrow_type = array_type  # For GEOGRAPHY, at least initially
        # is_nullable = bq_field.mode.upper() == "NULLABLE"
        is_nullable = True
```

## Exception if remove nullability

```
        if self._exception is not None:
            # pylint: disable=raising-bad-type
            # Pylint doesn't recognize that this is valid in this case.
>           raise self._exception
E           google.api_core.exceptions.BadRequest: 400 Required field phash cannot be null
```

other way to fix this is to verify if data has null when column has data type as required.  created the pull request for that. 
@Gaurang033 Thank you so much for your investigation, and proposing a solution! It looks like pyarrow did not check whether the arrays are consistent with schema when `Table.from_arrays()` is called, because `table = pa.Table.from_arrays(array, schema=pa.schema(fields))`(in your reproduction example) yields the following without an error: 
```
   id  name
0  10   AAA
1  11  None
2  12   BBB
3  13   CCC
```
It feels like a major neligence on the pyarrow side. I wonder if it's a bug, or it's done by design. 

At the same time, we have an umbrella issue https://github.com/googleapis/python-bigquery/issues/1646, which covers a bunch of issues related to the `load_table_from_*()` methods. We may need a more comprehensive look into them all, in case this patch would break other user paths.
@Linchin  totally agree. Seems like a bug from pyarrow library as I can reproduce it without using bigquery.  if you think we need this patch until pyarrow fix this, let's merge this. I fixed the failing test case.  when we provided unknown data type arrow field was coming as null. and so method I added was throwing exception. 
I added a similar comment on the PR #1698:

> I worry about spending too much time and energy trying to create a work around for what we all agree is a problem in pyarrow. This feels like it creates greater complexity in our code, increased fragility, and a higher maintenance burden in the long run. 

> Am I missing something?

I am inclined to close this issue as out of scope for us.
Given that arrow is intended for fast conversions, I think the correct fix for us is to stop using REQUIRED/NULLABLE from BigQuery to determine our local arrow schema. Instead, always set nullable=True locally.

Resolving patch:
diff --git a/google/cloud/bigquery/_pandas_helpers.py b/google/cloud/bigquery/_pandas_helpers.py
--- a/google/cloud/bigquery/_pandas_helpers.py
+++ b/google/cloud/bigquery/_pandas_helpers.py
@@ -178,12 +178,18 @@ def bq_to_arrow_field(bq_field, array_type=None):
     if arrow_type is not None:
         if array_type is not None:
             arrow_type = array_type  # For GEOGRAPHY, at least initially
-        is_nullable = bq_field.mode.upper() == "NULLABLE"
         metadata = BQ_FIELD_TYPE_TO_ARROW_FIELD_METADATA.get(
             bq_field.field_type.upper() if bq_field.field_type else ""
         )
         return pyarrow.field(
-            bq_field.name, arrow_type, nullable=is_nullable, metadata=metadata
+            bq_field.name,
+            arrow_type,
+            # Even if the remote schema is REQUIRED, there's a chance there's
+            # local NULL values. Arrow will gladly interpret these NULL values
+            # as non-NULL and give you an arbitrary value. See:
+            # https://github.com/googleapis/python-bigquery/issues/1692
+            nullable=True,
+            metadata=metadata,
         )
 
     warnings.warn("Unable to determine type for field '{}'.".format(bq_field.name))


Test patch:
diff --git a/tests/system/test_pandas.py b/tests/system/test_pandas.py
--- a/tests/system/test_pandas.py
+++ b/tests/system/test_pandas.py
@@ -428,8 +428,7 @@ def test_load_table_from_dataframe_w_nulls(bigquery_client, dataset_id):
 
 
 def test_load_table_from_dataframe_w_required(bigquery_client, dataset_id):
-    """Test that a DataFrame with required columns can be uploaded if a
-    BigQuery schema is specified.
+    """Test that a DataFrame can be uploaded to a table with required columns.
 
     See: https://github.com/googleapis/google-cloud-python/issues/8093
     """
@@ -440,7 +439,6 @@ def test_load_table_from_dataframe_w_required(bigquery_client, dataset_id):
 
     records = [{"name": "Chip", "age": 2}, {"name": "Dale", "age": 3}]
     dataframe = pandas.DataFrame(records, columns=["name", "age"])
-    job_config = bigquery.LoadJobConfig(schema=table_schema)
     table_id = "{}.{}.load_table_from_dataframe_w_required".format(
         bigquery_client.project, dataset_id
     )
@@ -451,15 +449,50 @@ def test_load_table_from_dataframe_w_required(bigquery_client, dataset_id):
         bigquery.Table(table_id, schema=table_schema)
     )
 
-    job_config = bigquery.LoadJobConfig(schema=table_schema)
-    load_job = bigquery_client.load_table_from_dataframe(
-        dataframe, table_id, job_config=job_config
-    )
+    load_job = bigquery_client.load_table_from_dataframe(dataframe, table_id)
     load_job.result()
 
     table = bigquery_client.get_table(table)
     assert tuple(table.schema) == table_schema
     assert table.num_rows == 2
+    for field in table.schema:
+        assert field.mode == "REQUIRED"
+
+
+def test_load_table_from_dataframe_w_required_but_local_nulls_fails(
+    bigquery_client, dataset_id
+):
+    """Test that a DataFrame with nulls can't be uploaded to a table with
+    required columns.
+
+    See: https://github.com/googleapis/python-bigquery/issues/1692
+    """
+    table_schema = (
+        bigquery.SchemaField("name", "STRING", mode="REQUIRED"),
+        bigquery.SchemaField("age", "INTEGER", mode="REQUIRED"),
+    )
+
+    records = [
+        {"name": "Chip", "age": 2},
+        {"name": "Dale", "age": 3},
+        {"name": None, "age": None},
+        {"name": "Alvin", "age": 4},
+    ]
+    dataframe = pandas.DataFrame(records, columns=["name", "age"])
+    table_id = (
+        "{}.{}.load_table_from_dataframe_w_required_but_local_nulls_fails".format(
+            bigquery_client.project, dataset_id
+        )
+    )
+
+    # Create the table before loading so that schema mismatch errors are
+    # identified.
+    helpers.retry_403(bigquery_client.create_table)(
+        bigquery.Table(table_id, schema=table_schema)
+    )
+
+    with pytest.raises(google.api_core.exceptions.BadRequest, match="null"):
+        bigquery_client.load_table_from_dataframe(dataframe, table_id).result()
 
 
 def test_load_table_from_dataframe_w_explicit_schema(bigquery_client, dataset_id):
diff --git a/tests/unit/test__pandas_helpers.py b/tests/unit/test__pandas_helpers.py
--- a/tests/unit/test__pandas_helpers.py
+++ b/tests/unit/test__pandas_helpers.py
@@ -1017,30 +1017,41 @@ def test_dataframe_to_arrow_with_required_fields(module_under_test):
     )
 
     data = {
-        "field01": ["hello", "world"],
-        "field02": [b"abd", b"efg"],
-        "field03": [1, 2],
-        "field04": [3, 4],
-        "field05": [1.25, 9.75],
-        "field06": [-1.75, -3.5],
-        "field07": [decimal.Decimal("1.2345"), decimal.Decimal("6.7891")],
+        "field01": ["hello", None, "world"],
+        "field02": [b"abd", b"efg", b"hij"],
+        "field03": [1, 2, 3],
+        "field04": [4, None, 5],
+        "field05": [1.25, 0.0, 9.75],
+        "field06": [-1.75, None, -3.5],
+        "field07": [
+            decimal.Decimal("1.2345"),
+            decimal.Decimal("6.7891"),
+            -decimal.Decimal("10.111213"),
+        ],
         "field08": [
             decimal.Decimal("-{d38}.{d38}".format(d38="9" * 38)),
+            None,
             decimal.Decimal("{d38}.{d38}".format(d38="9" * 38)),
         ],
-        "field09": [True, False],
-        "field10": [False, True],
+        "field09": [True, False, True],
+        "field10": [False, True, None],
         "field11": [
             datetime.datetime(1970, 1, 1, 0, 0, 0, tzinfo=datetime.timezone.utc),
             datetime.datetime(2012, 12, 21, 9, 7, 42, tzinfo=datetime.timezone.utc),
+            datetime.datetime(2022, 7, 14, 23, 59, 59, tzinfo=datetime.timezone.utc),
         ],
-        "field12": [datetime.date(9999, 12, 31), datetime.date(1970, 1, 1)],
-        "field13": [datetime.time(23, 59, 59, 999999), datetime.time(12, 0, 0)],
+        "field12": [datetime.date(9999, 12, 31), None, datetime.date(1970, 1, 1)],
+        "field13": [datetime.time(23, 59, 59, 999999), None, datetime.time(12, 0, 0)],
         "field14": [
             datetime.datetime(1970, 1, 1, 0, 0, 0),
+            None,
             datetime.datetime(2012, 12, 21, 9, 7, 42),
         ],
-        "field15": ["POINT(30 10)", "POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10))"],
+        "field15": [
+            None,
+            "POINT(30 10)",
+            "POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10))",
+        ],
     }
     dataframe = pandas.DataFrame(data)
 
@@ -1049,7 +1060,11 @@ def test_dataframe_to_arrow_with_required_fields(module_under_test):
 
     assert len(arrow_schema) == len(bq_schema)
     for arrow_field in arrow_schema:
-        assert not arrow_field.nullable
+        # Even if the remote schema is REQUIRED, there's a chance there's
+        # local NULL values. Arrow will gladly interpret these NULL values
+        # as non-NULL and give you an arbitrary value. See:
+        # https://github.com/googleapis/python-bigquery/issues/1692
+        assert arrow_field.nullable
 
 
 @pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
@@ -1101,7 +1116,11 @@ def test_dataframe_to_arrow_dict_sequence_schema(module_under_test):
     arrow_schema = arrow_table.schema
 
     expected_fields = [
-        pyarrow.field("field01", "string", nullable=False),
+        # Even if the remote schema is REQUIRED, there's a chance there's
+        # local NULL values. Arrow will gladly interpret these NULL values
+        # as non-NULL and give you an arbitrary value. See:
+        # https://github.com/googleapis/python-bigquery/issues/1692
+        pyarrow.field("field01", "string", nullable=True),
         pyarrow.field("field02", "bool", nullable=True),
     ]
     assert list(arrow_schema) == expected_fields