Repository: googleapis/python-bigquery

Issue:
Bug: `dataframe_to_json_generator` doesn't support `pandas.NA` type
#### Environment details

  - OS type and version: Ubuntu 20.04.2
  - Python version: `3.9.4`
  - pip version: `21.1.2`
  - `google-cloud-bigquery` version: `2.20.0`

#### Steps to reproduce

  1. Run the code example below

#### Code example

```python
import pandas
from google.cloud import bigquery

df = pandas.DataFrame({
    "series_a": [1, 2, pandas.NA]
})

json_iter = bigquery._pandas_helpers.dataframe_to_json_generator(df)
for row in json_iter:
    print(row)
```

#### Stack trace
```
{'series_a': 1}
{'series_a': 2}
Traceback (most recent call last):
  File "/home/christian/code/bug_example.py", line 11, in <module>
    for row in json_iter:
  File "/home/christian/miniconda3/envs/data-services-prod/lib/python3.9/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 783, in dataframe_to_json_generator
    if value != value:
  File "pandas/_libs/missing.pyx", line 360, in pandas._libs.missing.NAType.__bool__
TypeError: boolean value of NA is ambiguous
```

#### Suggested fix

Starting with pandas 1.0, an experimental pandas.NA value (singleton) is available to represent scalar missing values as
opposed to numpy.nan. Comparing the variable with itself (`value != value`) results in a `TypeError` as the `pandas.NA` value [doesn't support type-casting to boolean](https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#na-in-a-boolean-context).

I am planning to make a PR that switches the syntax `value != value` on [_pandas_helpers.py#L783](https://github.com/googleapis/python-bigquery/blob/ddd9ce714ebc496e22bc122e89f3bbee36ea15be/google/cloud/bigquery/_pandas_helpers.py#L783) to use the `pandas.isna` function but wanted to check if there is a better solution before I submit a patch?

Discussion:
plamut: FWIW, the `prerelease-deps` failure is flakiness, while Python 3.6 unit tests failure is legitimate - we use the oldest supported depenency versions there, and the new test failed with an attribute error:

```
 AttributeError: module 'pandas' has no attribute 'NA'
```
LinuxChristian: I added one commit per comment so it's easier to review. If you would like me to squash a final version let me know.
plamut: It appears that the test was not skipped?

```
    def test_dataframe_to_json_generator(module_under_test):
        utcnow = datetime.datetime.utcnow()
        df_data = collections.OrderedDict(
            [
>               ("a_series", [pandas.NA, 2, 3, 4]),
                ("b_series", [0.1, float("NaN"), 0.3, 0.4]),
                ("c_series", ["a", "b", pandas.NA, "d"]),
                ("d_series", [utcnow, utcnow, utcnow, pandas.NaT]),
                ("e_series", [True, False, True, None]),
            ]
        )
E       AttributeError: 'NoneType' object has no attribute 'NA'
```

Also reproducible locally.

Patch:
diff --git a/google/cloud/bigquery/_pandas_helpers.py b/google/cloud/bigquery/_pandas_helpers.py
index e93a99eb..285c0e83 100644
--- a/google/cloud/bigquery/_pandas_helpers.py
+++ b/google/cloud/bigquery/_pandas_helpers.py
@@ -780,7 +780,7 @@ def dataframe_to_json_generator(dataframe):
         output = {}
         for column, value in zip(dataframe.columns, row):
             # Omit NaN values.
-            if value != value:
+            if pandas.isna(value):
                 continue
             output[column] = value
         yield output


Test patch:
diff --git a/tests/unit/test__pandas_helpers.py b/tests/unit/test__pandas_helpers.py
index 39a3d845..aa87e28f 100644
--- a/tests/unit/test__pandas_helpers.py
+++ b/tests/unit/test__pandas_helpers.py
@@ -19,6 +19,7 @@ import functools
 import operator
 import queue
 import warnings
+import pkg_resources
 
 import mock
 
@@ -47,6 +48,14 @@ try:
 except ImportError:  # pragma: NO COVER
     bigquery_storage = None
 
+PANDAS_MINIUM_VERSION = pkg_resources.parse_version("1.0.0")
+
+if pandas is not None:
+    PANDAS_INSTALLED_VERSION = pkg_resources.get_distribution("pandas").parsed_version
+else:
+    # Set to less than MIN version.
+    PANDAS_INSTALLED_VERSION = pkg_resources.parse_version("0.0.0")
+
 
 skip_if_no_bignumeric = pytest.mark.skipif(
     not _BIGNUMERIC_SUPPORT, reason="BIGNUMERIC support requires pyarrow>=3.0.0",
@@ -734,6 +743,37 @@ def test_list_columns_and_indexes_with_named_index_same_as_column_name(
     assert columns_and_indexes == expected
 
 
+@pytest.mark.skipif(
+    pandas is None or PANDAS_INSTALLED_VERSION < PANDAS_MINIUM_VERSION,
+    reason="Requires `pandas version >= 1.0.0` which introduces pandas.NA",
+)
+def test_dataframe_to_json_generator(module_under_test):
+    utcnow = datetime.datetime.utcnow()
+    df_data = collections.OrderedDict(
+        [
+            ("a_series", [pandas.NA, 2, 3, 4]),
+            ("b_series", [0.1, float("NaN"), 0.3, 0.4]),
+            ("c_series", ["a", "b", pandas.NA, "d"]),
+            ("d_series", [utcnow, utcnow, utcnow, pandas.NaT]),
+            ("e_series", [True, False, True, None]),
+        ]
+    )
+    dataframe = pandas.DataFrame(
+        df_data, index=pandas.Index([4, 5, 6, 7], name="a_index")
+    )
+
+    dataframe = dataframe.astype({"a_series": pandas.Int64Dtype()})
+
+    rows = module_under_test.dataframe_to_json_generator(dataframe)
+    expected = [
+        {"b_series": 0.1, "c_series": "a", "d_series": utcnow, "e_series": True},
+        {"a_series": 2, "c_series": "b", "d_series": utcnow, "e_series": False},
+        {"a_series": 3, "b_series": 0.3, "d_series": utcnow, "e_series": True},
+        {"a_series": 4, "b_series": 0.4, "c_series": "d"},
+    ]
+    assert list(rows) == expected
+
+
 @pytest.mark.skipif(pandas is None, reason="Requires `pandas`")
 def test_list_columns_and_indexes_with_named_index(module_under_test):
     df_data = collections.OrderedDict(