Repository: googleapis/python-bigquery

Issue:
BigQuery: Upload pandas DataFrame containing arrays
The support for python Bigquery API indicates that arrays are possible, however, when passing from a pandas dataframe to bigquery there is a pyarrow struct issue.

The only way round it seems its to drop columns then use JSON Normalise for a separate table.

```python
from google.cloud import bigquery

project = 'lake'
client = bigquery.Client(credentials=credentials, project=project)
dataset_ref = client.dataset('XXX')
table_ref = dataset_ref.table('RAW_XXX')
job_config = bigquery.LoadJobConfig()
job_config.autodetect = True
job_config.write_disposition = 'WRITE_TRUNCATE'

client.load_table_from_dataframe(appended_data, table_ref,job_config=job_config).result()
```

This is the error recieved. NotImplementedError: struct

The reason I wanted to use this API as it indicates Nested Array support, which is perfect for our data lake in BQ but I assume this doesn't work?


Discussion:
Thanks for the report. Struct support is reported at https://github.com/googleapis/google-cloud-python/issues/8191, but I'll keep this issue open as a feature request for arrays of scalar types.
I am not able to repro this issue with dataframe containing arrays only. Its not giving any exception.

i can see an exception when we use dict(STRUCT/RECORD) in dataframe.

@AETDDraper  Could you please provide a sample dataframe containing arrays which causes the error.

google-cloud-bigquery 1.20.0
pyarrow 0.14.0
pandas 0.25.1
Loading a dataframe containing an array (a Python type List) does work if you don't specific the schema in the `LoadJobConfig()` and just let it detect it with `autodetect=True`. You just get a warning from pyarrow: `UserWarning: Pyarrow could not determine the type of columns: COLUMN_NAME.
  ", ".join(field.name for field in unknown_type_fields)`
```
df = pd.DataFrame([{'uuid':'test1', 'class':['cls1', 'cls1']}])
job_config = bigquery.LoadJobConfig(
	autodetect=True
)

bqclient.load_table_from_dataframe(
	df,
	"DATASET.TABLE",
	job_config=job_config
)
```
The detected schema looks like this:

field_name | type | mode
-- | -- | --
class | RECORD | NULLABLE
class.list | RECORD | REPEATED
class.list.item | STRING | NULLABLE

There is no way to provide a schema with the ARRAY type though, which is a bit frustrating. Or even RECORD or STRUCT type.

My config:
```
google-cloud-bigquery==1.24.0
pandas==1.0.1
pyarrow==0.16.0
```
> There is no way to provide a schema with the ARRAY type though, which is a bit frustrating. Or even RECORD or STRUCT type.

FWIW, uploading STRUCT columns through a dataframe will become possible when a fix in the `pyarrow` dependency will be released. I [tried](https://github.com/googleapis/python-bigquery/issues/21#issuecomment-606549404) that with the development `pyarrow` version and it worked.

At least uploading REPEATED fields (even those consisting of STRUCTs)  works with the `load_table_from_json()` method, if that workaround is feasible. I managed to successfully upload the following data with it:

```py
schema = [
    bigquery.SchemaField(
        "bar",
        "STRUCT",
        fields=[
            bigquery.SchemaField("aaa", "INTEGER", mode="REQUIRED"),
            bigquery.SchemaField("bbb", "INTEGER", mode="REQUIRED"),
        ],
        mode="REPEATED",
    ),
]

json_data = [
    {"bar": [{"aaa": 1, "bbb": 2}, {"aaa": 10, "bbb": 20}]},
    {"bar": [{"aaa": 3, "bbb": 4}, {"aaa": 5, "bbb": 6}]},
]

job_config = bigquery.LoadJobConfig(schema=schema)
client.load_table_from_json(json_data, table_ref, job_config=job_config).result()
```
As far as I can tell, the use case of loading a dataframe that contains an array (no struct) doesn't work even with the new version of Pyarrow (0.17.1).

Dataframe looks like this:
```
    field_A        field_B
0   1000135057     [-0.04591, -0.00742, ...
1   1100299862     [-0.15857, -0.17590, ...
2   1100174775     [0.075828, -0.1888, ...
3   1300002087     [0.097694, 0.06120, ...
4   1100190614     [0.097450, -0.10271, ...
```

Loading the dataframe:
```python
schema = [
    bigquery.SchemaField("field_A", "INTEGER"),
    bigquery.SchemaField("field_B", "FLOAT", "REPEATED"),
]

table = bigquery.Table(table_id, schema=schema)
bq_client.create_table(table, exists_ok=True)

job_config = bigquery.LoadJobConfig(
    schema=schema,
    destination=table_id,
    write_disposition="WRITE_TRUNCATE",
)
job = bq_client.load_table_from_dataframe(data, table_id, job_config=job_config)
job.result()
```

Produces:
>google.api_core.exceptions.BadRequest: 400 Incompatible type for field 'field_B'. Expected: 'FLOAT64', actual: 'STRUCT'.
@aaaaahaaaaa A use case of loading a dataframe that contains an array is not supported by bigquery and raised an error `Field is specified as REPEATED in provided schema which does not match REQUIRED as specified in the file`,  it is due to dataframe to parquest file conversion and for more information you can see https://github.com/googleapis/python-bigquery/issues/17 old issue related to that. 

Parquet limitation note for `REPEATED` fields has added and you could found in documentation of `load_table_from_dataframe` when `BigQuery's  1.26.0`  version will release.

Instead of using load_table_from_dataframe you can use load_table_from_json

example:
```python
    from google.cloud import bigquery
    import pandas
    client = bigquery.Client()

    schema = [bigquery.SchemaField("nested_repeated", "INTEGER", mode="REPEATED")]
    job_config = bigquery.LoadJobConfig(schema=schema)
    data = [{"nested_repeated": record}]
   
    client.load_table_from_json(data, "table_id", job_config=job_config).result()


So just to confirm my read here, am I correct in saying:

There is no way to use a `REPEATED` type with the `load_table_from_dataframe` function right now. This is due to how parquet serializes dataframes which is ultimately how this library serializes frames for transfer to bigquery. The current workaround is to use `load_table_from_json`. I put together a test to showcase how different methods breakdown here (run with code at master af2c987):

```
import sys

import pytest
import typing
import pandas as pd
from google.cloud.bigquery import Client, schema, job

T = typing.TypeVar("T")


def construct_repeated(
    lst: typing.List[T],
) -> typing.Mapping[str, typing.List[typing.Mapping[str, T]]]:
    """ when bigquery reads parquet arrays it ultimately structures the schema to look like this struct<list array<struct<item i>>>  """
    return {"list": [{"item": obj} for obj in lst]}


def destruct_repeated(
    dct: typing.Mapping[str, typing.List[typing.Mapping[str, T]]]
) -> typing.List[T]:
    return [d["item"] for d in dct["list"]]


def apply_col(
    col: str, callable: typing.Callable[[T], typing.Any]
) -> typing.Callable[[pd.DataFrame], pd.DataFrame]:
    def wrapper(df: pd.DataFrame) -> pd.DataFrame:
        df[col] = df[col].apply(callable)
        return df

    return wrapper


repeated_df = pd.DataFrame({"id": ["a", "b"], "targets": [[1], [2, 3]]})

repeated_schema = [
    schema.SchemaField("id", "STRING", "NULLABLE", None, (), None),
    schema.SchemaField("targets", "INTEGER", "REPEATED", None, (), None),
]

nested_df = repeated_df.copy()
nested_df["targets"] = nested_df["targets"].apply(construct_repeated)

nested_schema = [
    schema.SchemaField("id", "STRING", "NULLABLE", None, (), None),
    schema.SchemaField(
        "targets",
        "RECORD",
        "NULLABLE",
        None,
        (
            schema.SchemaField(
                "list",
                "RECORD",
                "REPEATED",
                None,
                (schema.SchemaField("item", "INTEGER", "NULLABLE", None, (), None),),
                None,
            ),
        ),
        None,
    ),
]


@pytest.fixture
def client():
    client = Client()
    yield client
    client.close()


@pytest.fixture
def dataset(client):
    d = client.create_dataset("test_dataset")
    yield d
    client.delete_dataset(d)


@pytest.fixture
def table(client, dataset):
    table = dataset.table("test_table")
    yield table
    client.delete_table(table, not_found_ok=True)


@pytest.mark.parametrize(
    "df, schema, post",
    [
        # FAILS WITH:
        # google.api_core.exceptions.BadRequest: 400 Error while reading data,
        # error message: Provided schema is not compatible with the file
        # 'prod-scotty-81de9f48-1a4f-424e-be99-2258b706fc46'. Field 'targets' is
        # specified as REPEATED in provided schema which does not match REQUIRED as
        # specified in the file.
        (repeated_df, repeated_schema, None),
        # FAILS comparing `df.to_dict() == got_back.to_dict()`:
        # doesn't upload structs properly, got_back ->
        # id targets
        # 0  a    None
        # 1  b    None
        (nested_df, nested_schema, None),
    ],
)
def test_roundtrip_from_dataframe(df, schema, post, client, table):
    client.load_table_from_dataframe(
        df, table, job_config=job.LoadJobConfig(schema=schema)
    ).result()

    got_back = client.list_rows(table).to_dataframe()
    if post is not None:
        got_back = post(got_back)

    assert df.to_dict() == got_back.to_dict()


@pytest.mark.parametrize(
    "df, schema, post",
    [
        # FAILS WITH:
        # google.api_core.exceptions.BadRequest: 400 Error while reading data,
        # error message: Provided schema is not compatible with the file
        # 'prod-scotty-81de9f48-1a4f-424e-be99-2258b706fc46'. Field 'targets' is
        # specified as REPEATED in provided schema which does not match REQUIRED as
        # specified in the file.
        (repeated_df, repeated_schema, None),
        # FAILS comparing `df.to_dict() == got_back.to_dict()`:
        # doesn't upload structs properly, got_back ->
        # id targets
        # 0  a    None
        # 1  b    None
        (nested_df, nested_schema, None),
    ],
)
def test_roundtrip_from_dataframe_existing_table(df, schema, post, client, table):
    table.schema = schema
    client.create_table(table)
    client.load_table_from_dataframe(df, table).result()

    got_back = client.list_rows(table).to_dataframe()
    if post is not None:
        got_back = post(got_back)

    assert df.to_dict() == got_back.to_dict()


@pytest.mark.parametrize(
    "df, schema, post",
    [
        # works but effectively applies construct_repeated on targets elements
        (repeated_df, repeated_schema, apply_col("targets", destruct_repeated)),
    ],
)
def test_roundtrip_from_dataframe_no_schema(df, schema, post, client, table):
    client.load_table_from_dataframe(df, table).result()
    got_back = client.list_rows(table).to_dataframe()

    if post is not None:
        got_back = post(got_back)

    assert df.to_dict() == got_back.to_dict()


@pytest.mark.parametrize(
    "df, schema, post",
    [
        # to_dataframe here returns an `np.array` type for lists which you can't compare
        # with a `list` type so we convert them for comparison
        (repeated_df, repeated_schema, apply_col("targets", list)),
        (nested_df, nested_schema, apply_col("targets", lambda d: {"list": list(d['list'])})),
    ],
)
def test_roundtrip_from_json(df, schema, post, client, table):
    client.load_table_from_json(
        df.to_dict(orient="records"), table, job_config=job.LoadJobConfig(schema=schema)
    ).result()

    got_back = client.list_rows(table).to_dataframe()
    if post is not None:
        got_back = post(got_back)

    assert df.to_dict() == got_back.to_dict()


if __name__ == "__main__":
    pytest.main([__file__, *sys.argv[1:]])
```

Pyarrow 2.0 released with improvements to the Parquet serialization. We should revisit to see if this issue can be resolved with pyarrow 2.0
@tswast  I have tried with these two following examples and it's working fine.

Example 1:
```python
import pandas as pd
import numpy as np
from google.cloud import bigquery

client = bigquery.Client()

job_config = bigquery.LoadJobConfig(autodetect=True)
table_id = bigquery.Table('table') 
table_id = client.create_table(table, exists_ok=True)
df = pd.DataFrame({'A': [np.array([1,2,3]), np.array([4,5,6]), np.array([7,8,9])]})
job = client.load_table_from_dataframe(df,"table_id",job_config=job_config).result()
```
Example 2:
```python
table_id_1 = bigquery.Table('table'_1) 
table_id_1= client.create_table(table, exists_ok=True)
job_config = bigquery.LoadJobConfig(autodetect=True)
df = pd.DataFrame([{'uuid':'test1', 'class':[['cls1','class2'], ['cls1','class3']]}])
job = client.load_table_from_dataframe(df,"table_id_1",job_config=job_config).result()
```
Let's add some system test cases and/or code samples for this (which are skipped unless we have pyarrow >=2.0) before we close this issue out.

I imagine it'd be useful to have samples which show more complex dataframes in the following docs:

* https://cloud.google.com/bigquery/docs/bigquery-storage-python-pandas
* https://cloud.google.com/bigquery/docs/pandas-gbq-migration
Based on the schema in #365, this feature is not yet supported. It still serializes to a strange format with an "item" column.
> Based on the schema in #365, this feature is not yet supported. It still serializes to a strange format with an "item" column.

We should look into how to integrate: [--parquet_enable_list_inference](https://cloud.google.com/bigquery/docs/reference/bq-cli-reference#parquet_enable_list_inference_flag) from BQ CLI
> > Based on the schema in #365, this feature is not yet supported. It still serializes to a strange format with an "item" column.
> 
> We should look into how to integrate: [--parquet_enable_list_inference](https://cloud.google.com/bigquery/docs/reference/bq-cli-reference#parquet_enable_list_inference_flag) from BQ CLI

We can already use that flag:
```python
parquet_options = bigquery.format_options.ParquetOptions()
parquet_options.enable_list_inference = True
job_config = bigquery.LoadJobConfig()
job_config.parquet_options = parquet_options
job = client.load_table_from_dataframe(
    df, TABLE_NAME, job_config=job_config
)
```
^ this is true but it only partly resolves this issue or at least the specific complaint "It still serializes to a strange format with an "item" column.")
```
client.load_table_from_dataframe(
    pd.DataFrame({"a": [[1, 2, 3], [3, 4, 5]]}), "tmp.repeated", job_config=job_config
)
client.list_rows(job.destination).to_dataframe()
```
`enable_list_inference=False`:
```
                                                   a
0  {'list': [{'item': 1}, {'item': 2}, {'item': 3}]}
1  {'list': [{'item': 3}, {'item': 4}, {'item': 5}]}
```
`enable_list_inference=True`:
```
                                         a
0  [{'item': 1}, {'item': 2}, {'item': 3}]
1  [{'item': 3}, {'item': 4}, {'item': 5}]
```
I think the probably has to do with implied nullability of the parquet schema (we would need to infer and remove nullness I think).
This works as a workaround:

```python
df = pd.DataFrame([[[1,2,3]], [[4,5,6]]], columns=('int64_array',))

writer = pyarrow.BufferOutputStream()
pyarrow.parquet.write_table(
    pyarrow.Table.from_pandas(df),
    writer,
    use_compliant_nested_type=True
)
reader = pyarrow.BufferReader(writer.getvalue())

client = bigquery.Client()
parquet_options = bigquery.format_options.ParquetOptions()
parquet_options.enable_list_inference = True
job_config = bigquery.LoadJobConfig()
job_config.source_format = bigquery.SourceFormat.PARQUET
job_config.parquet_options = parquet_options

job = client.load_table_from_file(
    reader, TABLE_NAME, job_config=job_config
)
```

I promise it does for realz this time.
@judahrand so the fix is using `use_compliant_nested_type=True`?
> @judahrand so the fix is using `use_compliant_nested_type=True`?

`use_compliant_nested_type=True` AND `parquet_options.enable_list_inference = True`. 

Both are needed. 
This is great, thanks @judahrand! Since as you point out the `list_inference` param is already exposed, I guess all that's needed to close this ticket is adding `use_compliant_nested_type=True` in https://github.com/googleapis/python-bigquery/blob/ba02f248ba9c449c34859579a4011f4bfd2f4a93/google/cloud/bigquery/_pandas_helpers.py#L521-L549 or allowing a way to configure it, depending on how breaking the change to that default would be
Yeah, exposing it somewhere would be the best - maybe don't change the default just document it 😛 

Only took 2+ years for an answer 🤣 
I'm happy to look into this if no one else fancies.

Resolving patch:
diff --git a/google/cloud/bigquery/_helpers.py b/google/cloud/bigquery/_helpers.py
--- a/google/cloud/bigquery/_helpers.py
+++ b/google/cloud/bigquery/_helpers.py
@@ -107,6 +107,9 @@ def verify_version(self):
 class PyarrowVersions:
     """Version comparisons for pyarrow package."""
 
+    # https://github.com/googleapis/python-bigquery/issues/781#issuecomment-883497414
+    _PYARROW_BAD_VERSIONS = frozenset([packaging.version.Version("2.0.0")])
+
     def __init__(self):
         self._installed_version = None
 
@@ -126,6 +129,14 @@ def installed_version(self) -> packaging.version.Version:
 
         return self._installed_version
 
+    @property
+    def is_bad_version(self) -> bool:
+        return self.installed_version in self._PYARROW_BAD_VERSIONS
+
+    @property
+    def use_compliant_nested_type(self) -> bool:
+        return self.installed_version.major >= 4
+
     def try_import(self, raise_if_error: bool = False) -> Any:
         """Verify that a recent enough version of pyarrow extra is
         installed.
diff --git a/google/cloud/bigquery/_pandas_helpers.py b/google/cloud/bigquery/_pandas_helpers.py
--- a/google/cloud/bigquery/_pandas_helpers.py
+++ b/google/cloud/bigquery/_pandas_helpers.py
@@ -79,8 +79,8 @@ def _to_wkb(v):
 _PANDAS_DTYPE_TO_BQ = {
     "bool": "BOOLEAN",
     "datetime64[ns, UTC]": "TIMESTAMP",
-    # BigQuery does not support uploading DATETIME values from Parquet files.
-    # See: https://github.com/googleapis/google-cloud-python/issues/9996
+    # TODO: Update to DATETIME in V3
+    # https://github.com/googleapis/python-bigquery/issues/985
     "datetime64[ns]": "TIMESTAMP",
     "float32": "FLOAT",
     "float64": "FLOAT",
@@ -396,7 +396,7 @@ def dataframe_to_bq_schema(dataframe, bq_schema):
     # column, but it was not found.
     if bq_schema_unused:
         raise ValueError(
-            u"bq_schema contains fields not present in dataframe: {}".format(
+            "bq_schema contains fields not present in dataframe: {}".format(
                 bq_schema_unused
             )
         )
@@ -405,7 +405,7 @@ def dataframe_to_bq_schema(dataframe, bq_schema):
     # pyarrow, if available.
     if unknown_type_fields:
         if not pyarrow:
-            msg = u"Could not determine the type of columns: {}".format(
+            msg = "Could not determine the type of columns: {}".format(
                 ", ".join(field.name for field in unknown_type_fields)
             )
             warnings.warn(msg)
@@ -444,7 +444,14 @@ def augment_schema(dataframe, current_bq_schema):
             continue
 
         arrow_table = pyarrow.array(dataframe[field.name])
-        detected_type = ARROW_SCALAR_IDS_TO_BQ.get(arrow_table.type.id)
+
+        if pyarrow.types.is_list(arrow_table.type):
+            # `pyarrow.ListType`
+            detected_mode = "REPEATED"
+            detected_type = ARROW_SCALAR_IDS_TO_BQ.get(arrow_table.values.type.id)
+        else:
+            detected_mode = field.mode
+            detected_type = ARROW_SCALAR_IDS_TO_BQ.get(arrow_table.type.id)
 
         if detected_type is None:
             unknown_type_fields.append(field)
@@ -453,7 +460,7 @@ def augment_schema(dataframe, current_bq_schema):
         new_field = schema.SchemaField(
             name=field.name,
             field_type=detected_type,
-            mode=field.mode,
+            mode=detected_mode,
             description=field.description,
             fields=field.fields,
         )
@@ -461,7 +468,7 @@ def augment_schema(dataframe, current_bq_schema):
 
     if unknown_type_fields:
         warnings.warn(
-            u"Pyarrow could not determine the type of columns: {}.".format(
+            "Pyarrow could not determine the type of columns: {}.".format(
                 ", ".join(field.name for field in unknown_type_fields)
             )
         )
@@ -500,7 +507,7 @@ def dataframe_to_arrow(dataframe, bq_schema):
     extra_fields = bq_field_names - column_and_index_names
     if extra_fields:
         raise ValueError(
-            u"bq_schema contains fields not present in dataframe: {}".format(
+            "bq_schema contains fields not present in dataframe: {}".format(
                 extra_fields
             )
         )
@@ -510,7 +517,7 @@ def dataframe_to_arrow(dataframe, bq_schema):
     missing_fields = column_names - bq_field_names
     if missing_fields:
         raise ValueError(
-            u"bq_schema is missing fields from dataframe: {}".format(missing_fields)
+            "bq_schema is missing fields from dataframe: {}".format(missing_fields)
         )
 
     arrow_arrays = []
@@ -530,7 +537,13 @@ def dataframe_to_arrow(dataframe, bq_schema):
     return pyarrow.Table.from_arrays(arrow_arrays, names=arrow_names)
 
 
-def dataframe_to_parquet(dataframe, bq_schema, filepath, parquet_compression="SNAPPY"):
+def dataframe_to_parquet(
+    dataframe,
+    bq_schema,
+    filepath,
+    parquet_compression="SNAPPY",
+    parquet_use_compliant_nested_type=True,
+):
     """Write dataframe as a Parquet file, according to the desired BQ schema.
 
     This function requires the :mod:`pyarrow` package. Arrow is used as an
@@ -551,14 +564,29 @@ def dataframe_to_parquet(dataframe, bq_schema, filepath, parquet_compression="SN
             The compression codec to use by the the ``pyarrow.parquet.write_table``
             serializing method. Defaults to "SNAPPY".
             https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html#pyarrow-parquet-write-table
+        parquet_use_compliant_nested_type (bool):
+            Whether the ``pyarrow.parquet.write_table`` serializing method should write
+            compliant Parquet nested type (lists). Defaults to ``True``.
+            https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#nested-types
+            https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html#pyarrow-parquet-write-table
+
+            This argument is ignored for ``pyarrow`` versions earlier than ``4.0.0``.
     """
     pyarrow = _helpers.PYARROW_VERSIONS.try_import(raise_if_error=True)
 
     import pyarrow.parquet
 
+    kwargs = (
+        {"use_compliant_nested_type": parquet_use_compliant_nested_type}
+        if _helpers.PYARROW_VERSIONS.use_compliant_nested_type
+        else {}
+    )
+
     bq_schema = schema._to_schema_fields(bq_schema)
     arrow_table = dataframe_to_arrow(dataframe, bq_schema)
-    pyarrow.parquet.write_table(arrow_table, filepath, compression=parquet_compression)
+    pyarrow.parquet.write_table(
+        arrow_table, filepath, compression=parquet_compression, **kwargs,
+    )
 
 
 def _row_iterator_page_to_arrow(page, column_names, arrow_types):
diff --git a/google/cloud/bigquery/client.py b/google/cloud/bigquery/client.py
--- a/google/cloud/bigquery/client.py
+++ b/google/cloud/bigquery/client.py
@@ -27,19 +27,11 @@
 import json
 import math
 import os
-import packaging.version
 import tempfile
 from typing import Any, BinaryIO, Dict, Iterable, Optional, Sequence, Tuple, Union
 import uuid
 import warnings
 
-try:
-    import pyarrow
-
-    _PYARROW_VERSION = packaging.version.parse(pyarrow.__version__)
-except ImportError:  # pragma: NO COVER
-    pyarrow = None
-
 from google import resumable_media  # type: ignore
 from google.resumable_media.requests import MultipartUpload
 from google.resumable_media.requests import ResumableUpload
@@ -103,6 +95,10 @@
 from google.cloud.bigquery.table import TableListItem
 from google.cloud.bigquery.table import TableReference
 from google.cloud.bigquery.table import RowIterator
+from google.cloud.bigquery.format_options import ParquetOptions
+from google.cloud.bigquery import _helpers
+
+pyarrow = _helpers.PYARROW_VERSIONS.try_import()
 
 
 _DEFAULT_CHUNKSIZE = 100 * 1024 * 1024  # 100 MB
@@ -128,8 +124,6 @@
 # https://github.com/googleapis/python-bigquery/issues/438
 _MIN_GET_QUERY_RESULTS_TIMEOUT = 120
 
-# https://github.com/googleapis/python-bigquery/issues/781#issuecomment-883497414
-_PYARROW_BAD_VERSIONS = frozenset([packaging.version.Version("2.0.0")])
 
 TIMEOUT_HEADER = "X-Server-Timeout"
 
@@ -2469,10 +2463,10 @@ def load_table_from_dataframe(
             They are supported when using the PARQUET source format, but
             due to the way they are encoded in the ``parquet`` file,
             a mismatch with the existing table schema can occur, so
-            100% compatibility cannot be guaranteed for REPEATED fields when
+            REPEATED fields are not properly supported when using ``pyarrow<4.0.0``
             using the parquet format.
 
-            https://github.com/googleapis/python-bigquery/issues/17
+            https://github.com/googleapis/python-bigquery/issues/19
 
         Args:
             dataframe (pandas.DataFrame):
@@ -2519,18 +2513,18 @@ def load_table_from_dataframe(
                 :attr:`~google.cloud.bigquery.job.SourceFormat.PARQUET` are
                 supported.
             parquet_compression (Optional[str]):
-                 [Beta] The compression method to use if intermittently
-                 serializing ``dataframe`` to a parquet file.
-
-                 The argument is directly passed as the ``compression``
-                 argument to the underlying ``pyarrow.parquet.write_table()``
-                 method (the default value "snappy" gets converted to uppercase).
-                 https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html#pyarrow-parquet-write-table
-
-                 If the job config schema is missing, the argument is directly
-                 passed as the ``compression`` argument to the underlying
-                 ``DataFrame.to_parquet()`` method.
-                 https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html#pandas.DataFrame.to_parquet
+                [Beta] The compression method to use if intermittently
+                serializing ``dataframe`` to a parquet file.
+
+                The argument is directly passed as the ``compression``
+                argument to the underlying ``pyarrow.parquet.write_table()``
+                method (the default value "snappy" gets converted to uppercase).
+                https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html#pyarrow-parquet-write-table
+
+                If the job config schema is missing, the argument is directly
+                passed as the ``compression`` argument to the underlying
+                ``DataFrame.to_parquet()`` method.
+                https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_parquet.html#pandas.DataFrame.to_parquet
             timeout (Optional[float]):
                 The number of seconds to wait for the underlying HTTP transport
                 before using ``retry``.
@@ -2562,6 +2556,16 @@ def load_table_from_dataframe(
         if job_config.source_format is None:
             # default value
             job_config.source_format = job.SourceFormat.PARQUET
+
+        if (
+            job_config.source_format == job.SourceFormat.PARQUET
+            and job_config.parquet_options is None
+        ):
+            parquet_options = ParquetOptions()
+            # default value
+            parquet_options.enable_list_inference = True
+            job_config.parquet_options = parquet_options
+
         if job_config.source_format not in supported_formats:
             raise ValueError(
                 "Got unexpected source_format: '{}'. Currently, only PARQUET and CSV are supported".format(
@@ -2628,12 +2632,12 @@ def load_table_from_dataframe(
         try:
 
             if job_config.source_format == job.SourceFormat.PARQUET:
-                if _PYARROW_VERSION in _PYARROW_BAD_VERSIONS:
+                if _helpers.PYARROW_VERSIONS.is_bad_version:
                     msg = (
                         "Loading dataframe data in PARQUET format with pyarrow "
-                        f"{_PYARROW_VERSION} can result in data corruption. It is "
-                        "therefore *strongly* advised to use a different pyarrow "
-                        "version or a different source format. "
+                        f"{_helpers.PYARROW_VERSIONS.installed_version} can result in data "
+                        "corruption. It is therefore *strongly* advised to use a "
+                        "different pyarrow version or a different source format. "
                         "See: https://github.com/googleapis/python-bigquery/issues/781"
                     )
                     warnings.warn(msg, category=RuntimeWarning)
@@ -2647,9 +2651,19 @@ def load_table_from_dataframe(
                         job_config.schema,
                         tmppath,
                         parquet_compression=parquet_compression,
+                        parquet_use_compliant_nested_type=True,
                     )
                 else:
-                    dataframe.to_parquet(tmppath, compression=parquet_compression)
+                    dataframe.to_parquet(
+                        tmppath,
+                        engine="pyarrow",
+                        compression=parquet_compression,
+                        **(
+                            {"use_compliant_nested_type": True}
+                            if _helpers.PYARROW_VERSIONS.use_compliant_nested_type
+                            else {}
+                        ),
+                    )
 
             else:
 


Test patch:
diff --git a/tests/system/test_pandas.py b/tests/system/test_pandas.py
--- a/tests/system/test_pandas.py
+++ b/tests/system/test_pandas.py
@@ -24,6 +24,7 @@
 import google.api_core.retry
 import pkg_resources
 import pytest
+import numpy
 
 from google.cloud import bigquery
 from . import helpers
@@ -84,6 +85,81 @@ def test_load_table_from_dataframe_w_automatic_schema(bigquery_client, dataset_i
             ("uint8_col", pandas.Series([0, 1, 2], dtype="uint8")),
             ("uint16_col", pandas.Series([3, 4, 5], dtype="uint16")),
             ("uint32_col", pandas.Series([6, 7, 8], dtype="uint32")),
+            ("array_bool_col", pandas.Series([[True], [False], [True]])),
+            (
+                "array_ts_col",
+                pandas.Series(
+                    [
+                        [
+                            datetime.datetime(
+                                2010, 1, 2, 3, 44, 50, tzinfo=datetime.timezone.utc
+                            ),
+                        ],
+                        [
+                            datetime.datetime(
+                                2011, 2, 3, 14, 50, 59, tzinfo=datetime.timezone.utc
+                            ),
+                        ],
+                        [
+                            datetime.datetime(
+                                2012, 3, 14, 15, 16, tzinfo=datetime.timezone.utc
+                            ),
+                        ],
+                    ],
+                ),
+            ),
+            (
+                "array_dt_col",
+                pandas.Series(
+                    [
+                        [datetime.datetime(2010, 1, 2, 3, 44, 50)],
+                        [datetime.datetime(2011, 2, 3, 14, 50, 59)],
+                        [datetime.datetime(2012, 3, 14, 15, 16)],
+                    ],
+                ),
+            ),
+            (
+                "array_float32_col",
+                pandas.Series(
+                    [numpy.array([_], dtype="float32") for _ in [1.0, 2.0, 3.0]]
+                ),
+            ),
+            (
+                "array_float64_col",
+                pandas.Series(
+                    [numpy.array([_], dtype="float64") for _ in [4.0, 5.0, 6.0]]
+                ),
+            ),
+            (
+                "array_int8_col",
+                pandas.Series(
+                    [numpy.array([_], dtype="int8") for _ in [-12, -11, -10]]
+                ),
+            ),
+            (
+                "array_int16_col",
+                pandas.Series([numpy.array([_], dtype="int16") for _ in [-9, -8, -7]]),
+            ),
+            (
+                "array_int32_col",
+                pandas.Series([numpy.array([_], dtype="int32") for _ in [-6, -5, -4]]),
+            ),
+            (
+                "array_int64_col",
+                pandas.Series([numpy.array([_], dtype="int64") for _ in [-3, -2, -1]]),
+            ),
+            (
+                "array_uint8_col",
+                pandas.Series([numpy.array([_], dtype="uint8") for _ in [0, 1, 2]]),
+            ),
+            (
+                "array_uint16_col",
+                pandas.Series([numpy.array([_], dtype="uint16") for _ in [3, 4, 5]]),
+            ),
+            (
+                "array_uint32_col",
+                pandas.Series([numpy.array([_], dtype="uint32") for _ in [6, 7, 8]]),
+            ),
         ]
     )
     dataframe = pandas.DataFrame(df_data, columns=df_data.keys())
@@ -99,9 +175,8 @@ def test_load_table_from_dataframe_w_automatic_schema(bigquery_client, dataset_i
     assert tuple(table.schema) == (
         bigquery.SchemaField("bool_col", "BOOLEAN"),
         bigquery.SchemaField("ts_col", "TIMESTAMP"),
-        # BigQuery does not support uploading DATETIME values from
-        # Parquet files. See:
-        # https://github.com/googleapis/google-cloud-python/issues/9996
+        # TODO: Update to DATETIME in V3
+        # https://github.com/googleapis/python-bigquery/issues/985
         bigquery.SchemaField("dt_col", "TIMESTAMP"),
         bigquery.SchemaField("float32_col", "FLOAT"),
         bigquery.SchemaField("float64_col", "FLOAT"),
@@ -112,6 +187,20 @@ def test_load_table_from_dataframe_w_automatic_schema(bigquery_client, dataset_i
         bigquery.SchemaField("uint8_col", "INTEGER"),
         bigquery.SchemaField("uint16_col", "INTEGER"),
         bigquery.SchemaField("uint32_col", "INTEGER"),
+        bigquery.SchemaField("array_bool_col", "BOOLEAN", mode="REPEATED"),
+        bigquery.SchemaField("array_ts_col", "TIMESTAMP", mode="REPEATED"),
+        # TODO: Update to DATETIME in V3
+        # https://github.com/googleapis/python-bigquery/issues/985
+        bigquery.SchemaField("array_dt_col", "TIMESTAMP", mode="REPEATED"),
+        bigquery.SchemaField("array_float32_col", "FLOAT", mode="REPEATED"),
+        bigquery.SchemaField("array_float64_col", "FLOAT", mode="REPEATED"),
+        bigquery.SchemaField("array_int8_col", "INTEGER", mode="REPEATED"),
+        bigquery.SchemaField("array_int16_col", "INTEGER", mode="REPEATED"),
+        bigquery.SchemaField("array_int32_col", "INTEGER", mode="REPEATED"),
+        bigquery.SchemaField("array_int64_col", "INTEGER", mode="REPEATED"),
+        bigquery.SchemaField("array_uint8_col", "INTEGER", mode="REPEATED"),
+        bigquery.SchemaField("array_uint16_col", "INTEGER", mode="REPEATED"),
+        bigquery.SchemaField("array_uint32_col", "INTEGER", mode="REPEATED"),
     )
     assert table.num_rows == 3
 
diff --git a/tests/unit/test_client.py b/tests/unit/test_client.py
--- a/tests/unit/test_client.py
+++ b/tests/unit/test_client.py
@@ -57,6 +57,7 @@
 from google.cloud import bigquery_v2
 from google.cloud.bigquery.dataset import DatasetReference
 from google.cloud.bigquery.retry import DEFAULT_TIMEOUT
+from google.cloud.bigquery import ParquetOptions
 
 try:
     from google.cloud import bigquery_storage
@@ -6942,6 +6943,179 @@ def test_load_table_from_dataframe_w_custom_job_config_w_source_format(self):
         # the original config object should not have been modified
         assert job_config.to_api_repr() == original_config_copy.to_api_repr()
 
+    @unittest.skipIf(pandas is None, "Requires `pandas`")
+    @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+    def test_load_table_from_dataframe_w_parquet_options_none(self):
+        from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+        from google.cloud.bigquery import job
+        from google.cloud.bigquery.schema import SchemaField
+
+        client = self._make_client()
+        records = [{"id": 1, "age": 100}, {"id": 2, "age": 60}]
+        dataframe = pandas.DataFrame(records)
+
+        job_config = job.LoadJobConfig(
+            write_disposition=job.WriteDisposition.WRITE_TRUNCATE,
+            source_format=job.SourceFormat.PARQUET,
+        )
+
+        get_table_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.get_table",
+            autospec=True,
+            return_value=mock.Mock(
+                schema=[SchemaField("id", "INTEGER"), SchemaField("age", "INTEGER")]
+            ),
+        )
+        load_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+        )
+        with load_patch as load_table_from_file, get_table_patch as get_table:
+            client.load_table_from_dataframe(
+                dataframe, self.TABLE_REF, job_config=job_config, location=self.LOCATION
+            )
+
+        # no need to fetch and inspect table schema for WRITE_TRUNCATE jobs
+        assert not get_table.called
+
+        load_table_from_file.assert_called_once_with(
+            client,
+            mock.ANY,
+            self.TABLE_REF,
+            num_retries=_DEFAULT_NUM_RETRIES,
+            rewind=True,
+            size=mock.ANY,
+            job_id=mock.ANY,
+            job_id_prefix=None,
+            location=self.LOCATION,
+            project=None,
+            job_config=mock.ANY,
+            timeout=DEFAULT_TIMEOUT,
+        )
+
+        sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+        assert sent_config.parquet_options.enable_list_inference is True
+
+    @unittest.skipIf(pandas is None, "Requires `pandas`")
+    @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+    def test_load_table_from_dataframe_w_list_inference_none(self):
+        from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+        from google.cloud.bigquery import job
+        from google.cloud.bigquery.schema import SchemaField
+
+        client = self._make_client()
+        records = [{"id": 1, "age": 100}, {"id": 2, "age": 60}]
+        dataframe = pandas.DataFrame(records)
+
+        parquet_options = ParquetOptions()
+
+        job_config = job.LoadJobConfig(
+            write_disposition=job.WriteDisposition.WRITE_TRUNCATE,
+            source_format=job.SourceFormat.PARQUET,
+        )
+        job_config.parquet_options = parquet_options
+
+        original_config_copy = copy.deepcopy(job_config)
+
+        get_table_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.get_table",
+            autospec=True,
+            return_value=mock.Mock(
+                schema=[SchemaField("id", "INTEGER"), SchemaField("age", "INTEGER")]
+            ),
+        )
+        load_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+        )
+        with load_patch as load_table_from_file, get_table_patch as get_table:
+            client.load_table_from_dataframe(
+                dataframe, self.TABLE_REF, job_config=job_config, location=self.LOCATION
+            )
+
+        # no need to fetch and inspect table schema for WRITE_TRUNCATE jobs
+        assert not get_table.called
+
+        load_table_from_file.assert_called_once_with(
+            client,
+            mock.ANY,
+            self.TABLE_REF,
+            num_retries=_DEFAULT_NUM_RETRIES,
+            rewind=True,
+            size=mock.ANY,
+            job_id=mock.ANY,
+            job_id_prefix=None,
+            location=self.LOCATION,
+            project=None,
+            job_config=mock.ANY,
+            timeout=DEFAULT_TIMEOUT,
+        )
+
+        sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+        assert sent_config.parquet_options.enable_list_inference is None
+
+        # the original config object should not have been modified
+        assert job_config.to_api_repr() == original_config_copy.to_api_repr()
+
+    @unittest.skipIf(pandas is None, "Requires `pandas`")
+    @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+    def test_load_table_from_dataframe_w_list_inference_false(self):
+        from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+        from google.cloud.bigquery import job
+        from google.cloud.bigquery.schema import SchemaField
+
+        client = self._make_client()
+        records = [{"id": 1, "age": 100}, {"id": 2, "age": 60}]
+        dataframe = pandas.DataFrame(records)
+
+        parquet_options = ParquetOptions()
+        parquet_options.enable_list_inference = False
+
+        job_config = job.LoadJobConfig(
+            write_disposition=job.WriteDisposition.WRITE_TRUNCATE,
+            source_format=job.SourceFormat.PARQUET,
+        )
+        job_config.parquet_options = parquet_options
+
+        original_config_copy = copy.deepcopy(job_config)
+
+        get_table_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.get_table",
+            autospec=True,
+            return_value=mock.Mock(
+                schema=[SchemaField("id", "INTEGER"), SchemaField("age", "INTEGER")]
+            ),
+        )
+        load_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+        )
+        with load_patch as load_table_from_file, get_table_patch as get_table:
+            client.load_table_from_dataframe(
+                dataframe, self.TABLE_REF, job_config=job_config, location=self.LOCATION
+            )
+
+        # no need to fetch and inspect table schema for WRITE_TRUNCATE jobs
+        assert not get_table.called
+
+        load_table_from_file.assert_called_once_with(
+            client,
+            mock.ANY,
+            self.TABLE_REF,
+            num_retries=_DEFAULT_NUM_RETRIES,
+            rewind=True,
+            size=mock.ANY,
+            job_id=mock.ANY,
+            job_id_prefix=None,
+            location=self.LOCATION,
+            project=None,
+            job_config=mock.ANY,
+            timeout=DEFAULT_TIMEOUT,
+        )
+
+        sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+        assert sent_config.parquet_options.enable_list_inference is False
+
+        # the original config object should not have been modified
+        assert job_config.to_api_repr() == original_config_copy.to_api_repr()
+
     @unittest.skipIf(pandas is None, "Requires `pandas`")
     @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
     def test_load_table_from_dataframe_w_custom_job_config_w_wrong_source_format(self):
@@ -7293,6 +7467,124 @@ def test_load_table_from_dataframe_struct_fields(self):
         assert sent_config.source_format == job.SourceFormat.PARQUET
         assert sent_config.schema == schema
 
+    @unittest.skipIf(pandas is None, "Requires `pandas`")
+    @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+    def test_load_table_from_dataframe_array_fields(self):
+        """Test that a DataFrame with array columns can be uploaded correctly.
+
+        See: https://github.com/googleapis/python-bigquery/issues/19
+        """
+        from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+        from google.cloud.bigquery import job
+        from google.cloud.bigquery.schema import SchemaField
+
+        client = self._make_client()
+
+        records = [(3.14, [1, 2])]
+        dataframe = pandas.DataFrame(
+            data=records, columns=["float_column", "array_column"]
+        )
+
+        schema = [
+            SchemaField("float_column", "FLOAT"),
+            SchemaField("array_column", "INTEGER", mode="REPEATED",),
+        ]
+        job_config = job.LoadJobConfig(schema=schema)
+
+        load_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+        )
+
+        get_table_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.get_table",
+            autospec=True,
+            side_effect=google.api_core.exceptions.NotFound("Table not found"),
+        )
+
+        with load_patch as load_table_from_file, get_table_patch:
+            client.load_table_from_dataframe(
+                dataframe,
+                self.TABLE_REF,
+                job_config=job_config,
+                location=self.LOCATION,
+            )
+
+        load_table_from_file.assert_called_once_with(
+            client,
+            mock.ANY,
+            self.TABLE_REF,
+            num_retries=_DEFAULT_NUM_RETRIES,
+            rewind=True,
+            size=mock.ANY,
+            job_id=mock.ANY,
+            job_id_prefix=None,
+            location=self.LOCATION,
+            project=None,
+            job_config=mock.ANY,
+            timeout=DEFAULT_TIMEOUT,
+        )
+
+        sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+        assert sent_config.source_format == job.SourceFormat.PARQUET
+        assert sent_config.schema == schema
+
+    @unittest.skipIf(pandas is None, "Requires `pandas`")
+    @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
+    def test_load_table_from_dataframe_array_fields_w_auto_schema(self):
+        """Test that a DataFrame with array columns can be uploaded correctly.
+
+        See: https://github.com/googleapis/python-bigquery/issues/19
+        """
+        from google.cloud.bigquery.client import _DEFAULT_NUM_RETRIES
+        from google.cloud.bigquery import job
+        from google.cloud.bigquery.schema import SchemaField
+
+        client = self._make_client()
+
+        records = [(3.14, [1, 2])]
+        dataframe = pandas.DataFrame(
+            data=records, columns=["float_column", "array_column"]
+        )
+
+        expected_schema = [
+            SchemaField("float_column", "FLOAT"),
+            SchemaField("array_column", "INT64", mode="REPEATED",),
+        ]
+
+        load_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+        )
+
+        get_table_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.get_table",
+            autospec=True,
+            side_effect=google.api_core.exceptions.NotFound("Table not found"),
+        )
+
+        with load_patch as load_table_from_file, get_table_patch:
+            client.load_table_from_dataframe(
+                dataframe, self.TABLE_REF, location=self.LOCATION,
+            )
+
+        load_table_from_file.assert_called_once_with(
+            client,
+            mock.ANY,
+            self.TABLE_REF,
+            num_retries=_DEFAULT_NUM_RETRIES,
+            rewind=True,
+            size=mock.ANY,
+            job_id=mock.ANY,
+            job_id_prefix=None,
+            location=self.LOCATION,
+            project=None,
+            job_config=mock.ANY,
+            timeout=DEFAULT_TIMEOUT,
+        )
+
+        sent_config = load_table_from_file.mock_calls[0][2]["job_config"]
+        assert sent_config.source_format == job.SourceFormat.PARQUET
+        assert sent_config.schema == expected_schema
+
     @unittest.skipIf(pandas is None, "Requires `pandas`")
     @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
     def test_load_table_from_dataframe_w_partial_schema(self):
@@ -7540,9 +7832,13 @@ def test_load_table_from_dataframe_w_bad_pyarrow_issues_warning(self):
         records = [{"id": 1, "age": 100}, {"id": 2, "age": 60}]
         dataframe = pandas.DataFrame(records)
 
+        _helpers_mock = mock.MagicMock()
+        _helpers_mock.PYARROW_VERSIONS = mock.MagicMock()
+        _helpers_mock.PYARROW_VERSIONS.installed_version = packaging.version.parse(
+            "2.0.0"
+        )  # A known bad version of pyarrow.
         pyarrow_version_patch = mock.patch(
-            "google.cloud.bigquery.client._PYARROW_VERSION",
-            packaging.version.parse("2.0.0"),  # A known bad version of pyarrow.
+            "google.cloud.bigquery.client._helpers", _helpers_mock
         )
         get_table_patch = mock.patch(
             "google.cloud.bigquery.client.Client.get_table",