Repository: googleapis/python-bigquery

Issue:
BUG: `bq_to_arrow_field` in `_pandas_helper.py`  always sets `pyarrow.field` to nullable


[Current implementation](https://github.com/googleapis/python-bigquery/blob/fabb2baebdc8a046c4fa00f9e1a65495430b927d/google/cloud/bigquery/_pandas_helpers.py#L203) of `bq_to_arrow_field` always sets `pyarrow.field` to nullable. However, since BigQuery array can not be set to nullable, the following error shows up when trying to upload a pandas dataframe to a bigquery table that contains a field in repeated mode. 


#### Stack trace
```
Traceback (most recent call last):

  File "/project_path/field_trie.py", line 426, in <module>

    main()

  File "/project_path/field_trie.py", line 363, in main

    job = client.load_table_from_dataframe(df, table)

          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/../lib/python3.12/site-packages/google/cloud/bigquery/client.py", line 2793, in load_table_from_dataframe

    _pandas_helpers.dataframe_to_parquet(

  File "/../lib/python3.12/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 669, in dataframe_to_parquet

    arrow_table = dataframe_to_arrow(dataframe, bq_schema)

                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/../lib/python3.12/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 611, in dataframe_to_arrow

    bq_to_arrow_array(get_column_or_index(dataframe, bq_field.name), bq_field)

  File "/../lib/python3.12/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 319, in bq_to_arrow_array

    return pyarrow.ListArray.from_pandas(series, type=arrow_type)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "pyarrow/array.pxi", line 1115, in pyarrow.lib.Array.from_pandas

  File "pyarrow/array.pxi", line 339, in pyarrow.lib.array

  File "pyarrow/array.pxi", line 85, in pyarrow.lib._ndarray_to_array

  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status

pyarrow.lib.ArrowInvalid: Cannot append scalar of type struct<order_id: int64, order_date: date32[day], event_timestamps: list<item: timestamp[us, tz=UTC]> not null, items: list<item: struct<item_id: int64, item_name: string, item_price: double>> not null> to builder for type struct<order_id: int64, order_date: date32[day], event_timestamps: list<item: timestamp[us, tz=UTC]>, items: list<item: struct<item_id: int64, item_name: string, item_price: double>>>
```


Discussion:
google-cla[bot]: Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this [failed invocation](https://github.com/googleapis/python-bigquery/pull/1999/checks?check_run_id=28887793544) of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Patch:
diff --git a/google/cloud/bigquery/_pandas_helpers.py b/google/cloud/bigquery/_pandas_helpers.py
index c21a0256..210ab487 100644
--- a/google/cloud/bigquery/_pandas_helpers.py
+++ b/google/cloud/bigquery/_pandas_helpers.py
@@ -200,7 +200,7 @@ def bq_to_arrow_field(bq_field, array_type=None):
             # local NULL values. Arrow will gladly interpret these NULL values
             # as non-NULL and give you an arbitrary value. See:
             # https://github.com/googleapis/python-bigquery/issues/1692
-            nullable=True,
+            nullable=False if bq_field.mode.upper() == "REPEATED" else True,
             metadata=metadata,
         )
 


Test patch:
diff --git a/tests/unit/test__pandas_helpers.py b/tests/unit/test__pandas_helpers.py
index 58d2b73b..203cc1d1 100644
--- a/tests/unit/test__pandas_helpers.py
+++ b/tests/unit/test__pandas_helpers.py
@@ -2002,6 +2002,23 @@ def test_bq_to_arrow_field_type_override(module_under_test):
     )
 
 
+@pytest.mark.skipif(isinstance(pyarrow, mock.Mock), reason="Requires `pyarrow`")
+def test_bq_to_arrow_field_set_repeated_nullable_false(module_under_test):
+    assert (
+        module_under_test.bq_to_arrow_field(
+            schema.SchemaField("name", "STRING", mode="REPEATED")
+        ).nullable
+        is False
+    )
+
+    assert (
+        module_under_test.bq_to_arrow_field(
+            schema.SchemaField("name", "STRING", mode="NULLABLE")
+        ).nullable
+        is True
+    )
+
+
 @pytest.mark.parametrize(
     "field_type, metadata",
     [