Repository: googleapis/google-cloud-python

Issue:
BigQuery: insert_rows() fails when a repeated field is missing
Using the method insert_rows to insert a row missing a REPEATED field returns an error. Repeated fields should be nullable by default, so it should not matter if a repeated field is not provided.
The method insert_rows_json, which insert_rows uses to make the API call seems to work fine.

#### Environment details
Python 2.7.15+
google-cloud-bigquery 1.21.0

#### Steps to reproduce
- Create a table with a repeated field
- Insert a row into the table using insert_rows, passing a dict where the repeated field key-value pair is missing.




Discussion:
@tswast AFAIK, this would be a back-end issue.
Seems related to #9207.
I was able to reproduce it in Python 3, too. Posting a minimal reproducible example (the table must already exist):
```py
from __future__ import print_function

from google.cloud import bigquery


PROJECT = "project_name"
DATASET = "dataset_name"
TABLE = "table_name"

client = bigquery.Client(PROJECT)
dataset_ref = client.dataset(DATASET)
table_ref = dataset_ref.table(TABLE)

schema = [
    bigquery.SchemaField("int_col", "INTEGER", mode="REQUIRED"),
    bigquery.SchemaField("repeated_col", "INTEGER", mode="REPEATED"),
])

rows = [
    # {"int_col": 5, "repeated_col": [10, 20, 30]},  # works
    # {"int_col": 8, "repeated_col": []},            # works 
    # {"int_col": 9, "repeated_col": None},          # "Field value cannot be empty."
    {"int_col": 12},                                 # "Field value cannot be empty."
]

print("Inserting rows...")
errors = client.insert_rows(table_ref, rows, selected_fields=schema)
print("Insert errors:,", errors)
```
The error is reproducible if the repeated field is not present in the data, or if it's set to `None` (even if `insert_rows_json()` is used directly). On the other hand, things work fine if the field is explicitly set to an empty list. All three cases should probably indeed be treated in the same way, though.

Classifying as P2, since a straightforward workaround exists.
Actually, we do have a client-side issue here. Inserting `{"int_col": 12}` works with `insert_rows_json`, but not `insert_rows`.

We need to update the logic for `insert_rows` to omit any missing values in the generated resource rather than the current logic, where it explicitly sets missing values to `None`.

https://github.com/googleapis/google-cloud-python/blob/b387134827dbc3be0e1b431201e0875798002fda/bigquery/google/cloud/bigquery/_helpers.py#L431

```
$ python write_python.py
----------------------------------------
insert_rows_json
----------------------------------------
Inserting populated rows...
Insert errors:

Inserting null rows...
Insert errors:
{'index': 0, 'errors': [{'reason': 'invalid', 'location': 'repeated_col', 'debugInfo': '', 'message': 'Field value cannot be empty.'}]}

Inserting missing rows...
Insert errors:

----------------------------------------
insert_rows
----------------------------------------
Inserting populated rows...
Insert errors:

Inserting null rows...
Insert errors:
{'index': 0, 'errors': [{'reason': 'invalid', 'location': 'repeated_col', 'debugInfo': '', 'message': 'Field value cannot be empty.'}]}

Inserting missing rows...
Insert errors:
{'index': 0, 'errors': [{'reason': 'invalid', 'location': 'repeated_col', 'debugInfo': '', 'message': 'Field value cannot be empty.'}]}
```

Source:

```python

from google.cloud import bigquery


client = bigquery.Client()

print("-" * 40)
print("insert_rows_json")
print("-" * 40)

rows = [
    {"repeated_col": []},  # works
    {"int_col": 5, "repeated_col": [10, 20, 30]},  # works
    {"int_col": 8, "repeated_col": []},            # works
]

print("Inserting populated rows...")
errors = client.insert_rows_json("my_dataset.google_cloud_python_9602", rows)
print("Insert errors:")
for error in errors:
    print(error)
print()

rows = [
    {"int_col": 9, "repeated_col": None},          # "Field value cannot be empty."
]
print("Inserting null rows...")
errors = client.insert_rows_json("my_dataset.google_cloud_python_9602", rows)
print("Insert errors:")
for error in errors:
    print(error)
print()

rows = [
    {"int_col": 12},                                 # works
]
print("Inserting missing rows...")
errors = client.insert_rows_json("my_dataset.google_cloud_python_9602", rows)
print("Insert errors:")
for error in errors:
    print(error)
print()


print("-" * 40)
print("insert_rows")
print("-" * 40)

schema = [
    bigquery.SchemaField("int_col", "INTEGER", mode="REQUIRED"),
    bigquery.SchemaField("repeated_col", "INTEGER", mode="REPEATED"),
]

rows = [
    {"repeated_col": []},  # works
    {"int_col": 5, "repeated_col": [10, 20, 30]},  # works
    {"int_col": 8, "repeated_col": []},            # works
]

print("Inserting populated rows...")
errors = client.insert_rows("my_dataset.google_cloud_python_9602", rows, selected_fields=schema)
print("Insert errors:")
for error in errors:
    print(error)
print()

rows = [
    {"int_col": 9, "repeated_col": None},          # "Field value cannot be empty."
]
print("Inserting null rows...")
errors = client.insert_rows("my_dataset.google_cloud_python_9602", rows, selected_fields=schema)
print("Insert errors:")
for error in errors:
    print(error)
print()

rows = [
    {"int_col": 12},                                 # "Field value cannot be empty."
]
print("Inserting missing rows...")
errors = client.insert_rows("my_dataset.google_cloud_python_9602", rows, selected_fields=schema)
print("Insert errors:")
for error in errors:
    print(error)
print()
```

As I don't believe the backend has changed it's behavior for this case, any backend issue I were to file would be considered a feature request.

Resolving patch:
diff --git a/bigquery/google/cloud/bigquery/_helpers.py b/bigquery/google/cloud/bigquery/_helpers.py
--- a/bigquery/google/cloud/bigquery/_helpers.py
+++ b/bigquery/google/cloud/bigquery/_helpers.py
@@ -424,11 +424,12 @@ def _record_field_to_json(fields, row_value):
 
     for subindex, subfield in enumerate(fields):
         subname = subfield.name
-        if isdict:
-            subvalue = row_value.get(subname)
-        else:
-            subvalue = row_value[subindex]
-        record[subname] = _field_to_json(subfield, subvalue)
+        subvalue = row_value.get(subname) if isdict else row_value[subindex]
+
+        # None values are unconditionally omitted
+        if subvalue is not None:
+            record[subname] = _field_to_json(subfield, subvalue)
+
     return record
 
 


Test patch:
diff --git a/bigquery/tests/unit/test__helpers.py b/bigquery/tests/unit/test__helpers.py
--- a/bigquery/tests/unit/test__helpers.py
+++ b/bigquery/tests/unit/test__helpers.py
@@ -856,14 +856,39 @@ def test_w_non_empty_dict(self):
         converted = self._call_fut(fields, original)
         self.assertEqual(converted, {"one": "42", "two": "two"})
 
-    def test_w_missing_nullable(self):
+    def test_w_some_missing_nullables(self):
         fields = [
             _make_field("INT64", name="one", mode="NULLABLE"),
             _make_field("STRING", name="two", mode="NULLABLE"),
         ]
         original = {"one": 42}
         converted = self._call_fut(fields, original)
-        self.assertEqual(converted, {"one": "42", "two": None})
+
+        # missing fields should not be converted to an explicit None
+        self.assertEqual(converted, {"one": "42"})
+
+    def test_w_all_missing_nullables(self):
+        fields = [
+            _make_field("INT64", name="one", mode="NULLABLE"),
+            _make_field("STRING", name="two", mode="NULLABLE"),
+        ]
+        original = {}
+        converted = self._call_fut(fields, original)
+
+        # we should get an empty dict, not None
+        self.assertEqual(converted, {})
+
+    def test_w_explicit_none_value(self):
+        fields = [
+            _make_field("INT64", name="one", mode="NULLABLE"),
+            _make_field("STRING", name="two", mode="NULLABLE"),
+            _make_field("BOOL", name="three", mode="REPEATED"),
+        ]
+        original = {"three": None, "one": 42, "two": None}
+        converted = self._call_fut(fields, original)
+
+        # None values should be dropped regardless of the field type
+        self.assertEqual(converted, {"one": "42"})
 
 
 class Test_field_to_json(unittest.TestCase):
diff --git a/bigquery/tests/unit/test_client.py b/bigquery/tests/unit/test_client.py
--- a/bigquery/tests/unit/test_client.py
+++ b/bigquery/tests/unit/test_client.py
@@ -4636,10 +4636,13 @@ def test_insert_rows_w_schema(self):
         ]
 
         def _row_data(row):
+            result = {"full_name": row[0], "age": str(row[1])}
             joined = row[2]
-            if isinstance(row[2], datetime.datetime):
+            if isinstance(joined, datetime.datetime):
                 joined = _microseconds_from_datetime(joined) * 1e-6
-            return {"full_name": row[0], "age": str(row[1]), "joined": joined}
+            if joined is not None:
+                result["joined"] = joined
+            return result
 
         SENT = {
             "rows": [
@@ -4708,7 +4711,10 @@ def test_insert_rows_w_list_of_dictionaries(self):
 
         def _row_data(row):
             joined = row["joined"]
-            if isinstance(joined, datetime.datetime):
+            if joined is None:
+                row = copy.deepcopy(row)
+                del row["joined"]
+            elif isinstance(joined, datetime.datetime):
                 row["joined"] = _microseconds_from_datetime(joined) * 1e-6
             row["age"] = str(row["age"])
             return row
@@ -4927,9 +4933,8 @@ def test_insert_rows_w_repeated_fields(self):
                 },
                 {
                     "json": {
-                        "color": None,
                         "items": [],
-                        "structs": [{"score": None, "times": [], "distances": [3.5]}],
+                        "structs": [{"times": [], "distances": [3.5]}],
                     },
                     "insertId": "1",
                 },
@@ -4996,10 +5001,7 @@ def test_insert_rows_w_record_schema(self):
                     },
                     "insertId": "1",
                 },
-                {
-                    "json": {"full_name": "Wylma Phlyntstone", "phone": None},
-                    "insertId": "2",
-                },
+                {"json": {"full_name": "Wylma Phlyntstone"}, "insertId": "2"},
             ]
         }