Repository: googleapis/python-bigquery

Issue:
load_table_from_dataframe produces incorrect results when used in list of dict 
#### Environment details

  - OS type and version: Debian 10 (dataproc image 2.0-debian10)
  - Python version: `python --version`: Python 3.8.10
  - pip version: `pip --version`: pip 21.1.2
  - `google-cloud-bigquery` version: `pip show google-cloud-bigquery`: google-cloud-bigquery==2.6.2, pyarrow==2.0.0

#### Steps to reproduce

  1. Create a big dataframe (1000 lines) with a column containing a list (at least length 6) of identically structured dictionaries 
  2. Create a bq client and use load_table_from_dataframe to create a table in bigquery
  3. Check the resulting table in bigquery. Structures seem to switch values with other instances in the list. (eg should have [STRUCT('w0' AS name, 0.1 AS value),STRUCT('h1' AS name, 1.2 AS value)] but have [STRUCT('h1' AS name, 0.1 AS value),STRUCT('w0' AS name, 1.2 AS value)]. The big problem is not the order, is that the integrity of information of each structure is not kept (eg. 'w0' should be 0.1, not 1.2). 

#### Code example

```python
# create df with a list of dictionaries
# In this example, the dict structure is {"name": str, "value":float}. name is a letter + int, and value are increasingly big floats
data = [[[{'name':'whyist'[i]+str(i), 'value':np.random.random()*10**i} for i in range(6)]] for n in range(1000)]
df = pd.DataFrame(data, columns=['vals'])

# load
project = 'myproject'
bq_client = bigquery.Client(project=project)
job_config = bigquery.LoadJobConfig()
job_config.write_disposition = 'WRITE_TRUNCATE'
bq_client.load_table_from_dataframe(
    dataframe=df,
    destination='tmp.test_bug',
    job_config = job_config
)

# Checking in bigquery, 
```
At least for this example, the 'value' attribute is transcribed in the correct order (first item has the smallest value, and it increases). The 'name' value was sampled with possibility of repetition. All table lines have the same 'name' values in the same order, and it can change if the code is reexecuted.



Discussion:
@Lmmejia11 Thanks for the runnable example, it appears `pyarrow` version is the culprit. This is the result I got with `pyarrow==4.0.0`, seemed fine to me (please confirm):

![Screenshot from 2021-07-20 17-33-41](https://user-images.githubusercontent.com/1235663/126352948-42c25338-bd44-4548-9e91-ab68e94fc5d8.png)

This was both with the latest BigQuery client (`v2.22.0`) as well as with the `v2.6.2` version.

However, when I downgraded `pyarrow` to `v2.0.0`, I started to see the names and values scrambled:

![Screenshot from 2021-07-20 17-41-52](https://user-images.githubusercontent.com/1235663/126354146-fd008332-1927-4608-ae8f-154b676dd349.png)

Would it be possible to upgrade `pyarrow` in your system?

I will nevertheless try to find why this is the case, because `pyarrow` is pinned to `>= 1.0.0`.

---

**Edit:** Interesting, this seems to start happening when the number of dataframe rows is 513 or more, while with 512 rows or less it works just fine. On the other hand, `pyarrow==4.0.0` works even with a large number of rows (at least up to 1 million, didn't test with more).

**Edit 2:** After additional tests it actually seems that `pyarrow==1.0.x` is also OK to use, only the `2.0.0` release causes problems.
Since we want to keep the supported dependency ranges wide (`pyarrow` can be tricky to upgrade), it appears that the most sensible thing would be detecting the `pyarrow` version at runtime in `load_table_from-dataframe()` and issue a warning if uploading more than 512 rows.

Also increasing priority, as possible silent data corruption is Bad™.
Thanks! thankfully I can upgrade pyarrow
As you said, I noticed the bug appeared when there are many lines. But it can also depend on the length of the lists. If I remember correctly, 1000 lines with list of length 5 also worked fine. I dont know if it could bug with less than 512 rows if the lists are longer or the values heavier, it might be linked to space.
Thanks, I'll keep this in mind. It might actually be better to _not_ try to narrow down the conditions when the bug could occur, but instead always issue a warning when a less recent `pyarrow` version is detected.

Resolving patch:
diff --git a/google/cloud/bigquery/client.py b/google/cloud/bigquery/client.py
--- a/google/cloud/bigquery/client.py
+++ b/google/cloud/bigquery/client.py
@@ -27,6 +27,7 @@
 import json
 import math
 import os
+import packaging.version
 import tempfile
 from typing import Any, BinaryIO, Dict, Iterable, Optional, Sequence, Tuple, Union
 import uuid
@@ -34,6 +35,8 @@
 
 try:
     import pyarrow
+
+    _PYARROW_VERSION = packaging.version.parse(pyarrow.__version__)
 except ImportError:  # pragma: NO COVER
     pyarrow = None
 
@@ -118,6 +121,9 @@
 # https://github.com/googleapis/python-bigquery/issues/438
 _MIN_GET_QUERY_RESULTS_TIMEOUT = 120
 
+# https://github.com/googleapis/python-bigquery/issues/781#issuecomment-883497414
+_PYARROW_BAD_VERSIONS = frozenset([packaging.version.Version("2.0.0")])
+
 
 class Project(object):
     """Wrapper for resource describing a BigQuery project.
@@ -2609,6 +2615,15 @@ def load_table_from_dataframe(
         try:
 
             if job_config.source_format == job.SourceFormat.PARQUET:
+                if _PYARROW_VERSION in _PYARROW_BAD_VERSIONS:
+                    msg = (
+                        "Loading dataframe data in PARQUET format with pyarrow "
+                        f"{_PYARROW_VERSION} can result in data corruption. It is "
+                        "therefore *strongly* advised to use a different pyarrow "
+                        "version or a different source format. "
+                        "See: https://github.com/googleapis/python-bigquery/issues/781"
+                    )
+                    warnings.warn(msg, category=RuntimeWarning)
 
                 if job_config.schema:
                     if parquet_compression == "snappy":  # adjust the default value


Test patch:
diff --git a/tests/unit/test_client.py b/tests/unit/test_client.py
--- a/tests/unit/test_client.py
+++ b/tests/unit/test_client.py
@@ -27,6 +27,7 @@
 import warnings
 
 import mock
+import packaging
 import requests
 import pytest
 import pytz
@@ -7510,6 +7511,42 @@ def test_load_table_from_dataframe_wo_pyarrow_raises_error(self):
                     parquet_compression="gzip",
                 )
 
+    def test_load_table_from_dataframe_w_bad_pyarrow_issues_warning(self):
+        pytest.importorskip("pandas", reason="Requires `pandas`")
+        pytest.importorskip("pyarrow", reason="Requires `pyarrow`")
+
+        client = self._make_client()
+        records = [{"id": 1, "age": 100}, {"id": 2, "age": 60}]
+        dataframe = pandas.DataFrame(records)
+
+        pyarrow_version_patch = mock.patch(
+            "google.cloud.bigquery.client._PYARROW_VERSION",
+            packaging.version.parse("2.0.0"),  # A known bad version of pyarrow.
+        )
+        get_table_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.get_table",
+            autospec=True,
+            side_effect=google.api_core.exceptions.NotFound("Table not found"),
+        )
+        load_patch = mock.patch(
+            "google.cloud.bigquery.client.Client.load_table_from_file", autospec=True
+        )
+
+        with load_patch, get_table_patch, pyarrow_version_patch:
+            with warnings.catch_warnings(record=True) as warned:
+                client.load_table_from_dataframe(
+                    dataframe, self.TABLE_REF, location=self.LOCATION,
+                )
+
+        expected_warnings = [
+            warning for warning in warned if "pyarrow" in str(warning).lower()
+        ]
+        assert len(expected_warnings) == 1
+        assert issubclass(expected_warnings[0].category, RuntimeWarning)
+        msg = str(expected_warnings[0].message)
+        assert "pyarrow 2.0.0" in msg
+        assert "data corruption" in msg
+
     @unittest.skipIf(pandas is None, "Requires `pandas`")
     @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
     def test_load_table_from_dataframe_w_nulls(self):