Repository: googleapis/python-bigquery

Issue:
add support for ParquetOptions for load/external tables
This FR is to add support for some advanced Parquet options (list inference and enums) when loading data or defining an external table.

Additional information has been shared out of band with library maintainers.

Discussion:
google-cla[bot]: We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors.  If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA ([login here](https://cla.developers.google.com/) to double check)?  If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment `@googlebot I fixed it.`. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2Fgoogleapis%2Fpython-bigquery%2Fpull%2F679) for more info**.

<!-- need_author_cla -->
plamut: We still need to expose `ParquetOptions` in the `google.cloud.bigquery` namespace to be consistent with `enums`, etc.
tseaver: @googlebot I fixed it.
google-cla[bot]: We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors.  If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA ([login here](https://cla.developers.google.com/) to double check)?  If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment `@googlebot I fixed it.`. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2Fgoogleapis%2Fpython-bigquery%2Fpull%2F679) for more info**.

<!-- need_author_cla -->
google-cla[bot]: We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors.  If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA ([login here](https://cla.developers.google.com/) to double check)?  If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment `@googlebot I fixed it.`. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2Fgoogleapis%2Fpython-bigquery%2Fpull%2F679) for more info**.

<!-- need_author_cla -->
plamut: Exposed the config object in `google.cloud.bigquery` namespace and also made sure that `parquet_options` works well with the `options` attribute.
simonvanderveldt: @plamut We are currently a bit blocked because this functionality is missing from the bigquery Python library. Is there anything we can do to help this get merged and released? Maybe some testing?
plamut: @simonvanderveldt Thanks for bumping this, it appears that it's incorrectly blocked by the CLA bot (and still needs one approval).

@shollyman Can you please convince the bot to change its mind? :) And add a review, if possible?

> Is there anything we can do to help this get merged and released? Maybe some testing?

If you want, you can test the BigQuery client with this PR branch, but primarily someone from Google with sufficient permissions needs to unblock the PR first.


simonvanderveldt: > If you want, you can test the BigQuery client with this PR branch, but primarily someone from Google with sufficient permissions needs to unblock the PR first.

Just tested it, working as expected :) We only need the list inference and using the library from this PR gives the same results as when using `bq load --parquet_enable_list_inference`.

I posted one review comment on the code btw.
googlebot: We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors.  If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA ([login here](https://cla.developers.google.com/) to double check)?  If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and then comment `@googlebot I fixed it.`. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2Fgoogleapis%2Fpython-bigquery%2Fpull%2F679) for more info**.

<!-- need_author_cla -->
stephaniewang526: Looks like we're missing CLA signing from `tseaver@palladion.com`
plamut: But Tres is a well-known contributor? Or is it because a different email address was used?
busunkim96: @plamut I looks like there is no CLA for `tseaver@palladion.com`. @tseaver's CLA lists `tres.seaver@gmail.com`.

I think if you edit the `Co-authored-by: Tres Seaver <tseaver@palladion.com>` in the commit body the CLA bot will be appeased. Please holler if that doesn't work.

google-cla[bot]: CLAs look good, thanks!

ℹ️ **Googlers: [Go here](https://goto.google.com/prinfo/https%3A%2F%2Fgithub.com%2Fgoogleapis%2Fpython-bigquery%2Fpull%2F679) for more info**.

<!-- ok -->
plamut: That trick with editing the commit message worked, great! Since we also have an approval now, I'll merge and release this today.
plamut: @busunkim96  As for the CLA bot, isn't this a bit fragile? Theoretically one could take another person's work posted as a suggestion, edit out the line in the commit message linking to that person, and then merge a non-CLA code into the project?

We can discuss this offline.

Patch:
diff --git a/google/cloud/bigquery/__init__.py b/google/cloud/bigquery/__init__.py
index ec08b2c8..f031cd81 100644
--- a/google/cloud/bigquery/__init__.py
+++ b/google/cloud/bigquery/__init__.py
@@ -47,6 +47,7 @@ from google.cloud.bigquery.external_config import BigtableColumn
 from google.cloud.bigquery.external_config import CSVOptions
 from google.cloud.bigquery.external_config import GoogleSheetsOptions
 from google.cloud.bigquery.external_config import ExternalSourceFormat
+from google.cloud.bigquery.format_options import ParquetOptions
 from google.cloud.bigquery.job import Compression
 from google.cloud.bigquery.job import CopyJob
 from google.cloud.bigquery.job import CopyJobConfig
@@ -136,6 +137,7 @@ __all__ = [
     "BigtableColumn",
     "CSVOptions",
     "GoogleSheetsOptions",
+    "ParquetOptions",
     "DEFAULT_RETRY",
     # Enum Constants
     "enums",
diff --git a/google/cloud/bigquery/external_config.py b/google/cloud/bigquery/external_config.py
index ef4d569f..0c49d2d7 100644
--- a/google/cloud/bigquery/external_config.py
+++ b/google/cloud/bigquery/external_config.py
@@ -27,6 +27,7 @@ from google.cloud.bigquery._helpers import _to_bytes
 from google.cloud.bigquery._helpers import _bytes_to_json
 from google.cloud.bigquery._helpers import _int_or_none
 from google.cloud.bigquery._helpers import _str_or_none
+from google.cloud.bigquery.format_options import ParquetOptions
 from google.cloud.bigquery.schema import SchemaField
 
 
@@ -53,6 +54,12 @@ class ExternalSourceFormat(object):
     DATASTORE_BACKUP = "DATASTORE_BACKUP"
     """Specifies datastore backup format"""
 
+    ORC = "ORC"
+    """Specifies ORC format."""
+
+    PARQUET = "PARQUET"
+    """Specifies Parquet format."""
+
     BIGTABLE = "BIGTABLE"
     """Specifies Bigtable format."""
 
@@ -540,7 +547,7 @@ class GoogleSheetsOptions(object):
         return config
 
 
-_OPTION_CLASSES = (BigtableOptions, CSVOptions, GoogleSheetsOptions)
+_OPTION_CLASSES = (BigtableOptions, CSVOptions, GoogleSheetsOptions, ParquetOptions)
 
 
 class HivePartitioningOptions(object):
@@ -784,6 +791,25 @@ class ExternalConfig(object):
             prop = {"fields": [field.to_api_repr() for field in value]}
         self._properties["schema"] = prop
 
+    @property
+    def parquet_options(self):
+        """Optional[google.cloud.bigquery.format_options.ParquetOptions]: Additional
+        properties to set if ``sourceFormat`` is set to PARQUET.
+
+        See:
+        https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#ExternalDataConfiguration.FIELDS.parquet_options
+        """
+        if self.source_format != ExternalSourceFormat.PARQUET:
+            return None
+        return self._options
+
+    @parquet_options.setter
+    def parquet_options(self, value):
+        if self.source_format != ExternalSourceFormat.PARQUET:
+            msg = f"Cannot set Parquet options, source format is {self.source_format}"
+            raise TypeError(msg)
+        self._options = value
+
     def to_api_repr(self) -> dict:
         """Build an API representation of this object.
 
diff --git a/google/cloud/bigquery/format_options.py b/google/cloud/bigquery/format_options.py
new file mode 100644
index 00000000..2c9a2ce2
--- /dev/null
+++ b/google/cloud/bigquery/format_options.py
@@ -0,0 +1,80 @@
+# Copyright 2021 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import copy
+from typing import Dict
+
+
+class ParquetOptions:
+    """Additional options if the PARQUET source format is used."""
+
+    _SOURCE_FORMAT = "PARQUET"
+    _RESOURCE_NAME = "parquetOptions"
+
+    def __init__(self):
+        self._properties = {}
+
+    @property
+    def enum_as_string(self) -> bool:
+        """Indicates whether to infer Parquet ENUM logical type as STRING instead of
+        BYTES by default.
+
+        See
+        https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#ParquetOptions.FIELDS.enum_as_string
+        """
+        return self._properties.get("enumAsString")
+
+    @enum_as_string.setter
+    def enum_as_string(self, value: bool) -> None:
+        self._properties["enumAsString"] = value
+
+    @property
+    def enable_list_inference(self) -> bool:
+        """Indicates whether to use schema inference specifically for Parquet LIST
+        logical type.
+
+        See
+        https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#ParquetOptions.FIELDS.enable_list_inference
+        """
+        return self._properties.get("enableListInference")
+
+    @enable_list_inference.setter
+    def enable_list_inference(self, value: bool) -> None:
+        self._properties["enableListInference"] = value
+
+    @classmethod
+    def from_api_repr(cls, resource: Dict[str, bool]) -> "ParquetOptions":
+        """Factory: construct an instance from a resource dict.
+
+        Args:
+            resource (Dict[str, bool]):
+                Definition of a :class:`~.format_options.ParquetOptions` instance in
+                the same representation as is returned from the API.
+
+        Returns:
+            :class:`~.format_options.ParquetOptions`:
+                Configuration parsed from ``resource``.
+        """
+        config = cls()
+        config._properties = copy.deepcopy(resource)
+        return config
+
+    def to_api_repr(self) -> dict:
+        """Build an API representation of this object.
+
+        Returns:
+            Dict[str, bool]:
+                A dictionary in the format used by the BigQuery API.
+        """
+        return copy.deepcopy(self._properties)
diff --git a/google/cloud/bigquery/job/load.py b/google/cloud/bigquery/job/load.py
index b8174af3..41d38dd7 100644
--- a/google/cloud/bigquery/job/load.py
+++ b/google/cloud/bigquery/job/load.py
@@ -16,6 +16,7 @@
 
 from google.cloud.bigquery.encryption_configuration import EncryptionConfiguration
 from google.cloud.bigquery.external_config import HivePartitioningOptions
+from google.cloud.bigquery.format_options import ParquetOptions
 from google.cloud.bigquery import _helpers
 from google.cloud.bigquery.schema import SchemaField
 from google.cloud.bigquery.schema import _to_schema_fields
@@ -439,6 +440,26 @@ class LoadJobConfig(_JobConfig):
     def write_disposition(self, value):
         self._set_sub_prop("writeDisposition", value)
 
+    @property
+    def parquet_options(self):
+        """Optional[google.cloud.bigquery.format_options.ParquetOptions]: Additional
+            properties to set if ``sourceFormat`` is set to PARQUET.
+
+        See:
+        https://cloud.google.com/bigquery/docs/reference/rest/v2/Job#JobConfigurationLoad.FIELDS.parquet_options
+        """
+        prop = self._get_sub_prop("parquetOptions")
+        if prop is not None:
+            prop = ParquetOptions.from_api_repr(prop)
+        return prop
+
+    @parquet_options.setter
+    def parquet_options(self, value):
+        if value is not None:
+            self._set_sub_prop("parquetOptions", value.to_api_repr())
+        else:
+            self._del_sub_prop("parquetOptions")
+
 
 class LoadJob(_AsyncJob):
     """Asynchronous job for loading data into a table.


Test patch:
diff --git a/tests/unit/job/test_load_config.py b/tests/unit/job/test_load_config.py
index 63f15ec5..b0729e42 100644
--- a/tests/unit/job/test_load_config.py
+++ b/tests/unit/job/test_load_config.py
@@ -700,3 +700,38 @@ class TestLoadJobConfig(_Base):
         self.assertEqual(
             config._properties["load"]["writeDisposition"], write_disposition
         )
+
+    def test_parquet_options_missing(self):
+        config = self._get_target_class()()
+        self.assertIsNone(config.parquet_options)
+
+    def test_parquet_options_hit(self):
+        config = self._get_target_class()()
+        config._properties["load"]["parquetOptions"] = dict(
+            enumAsString=True, enableListInference=False
+        )
+        self.assertTrue(config.parquet_options.enum_as_string)
+        self.assertFalse(config.parquet_options.enable_list_inference)
+
+    def test_parquet_options_setter(self):
+        from google.cloud.bigquery.format_options import ParquetOptions
+
+        parquet_options = ParquetOptions.from_api_repr(
+            dict(enumAsString=False, enableListInference=True)
+        )
+        config = self._get_target_class()()
+
+        config.parquet_options = parquet_options
+        self.assertEqual(
+            config._properties["load"]["parquetOptions"],
+            {"enumAsString": False, "enableListInference": True},
+        )
+
+    def test_parquet_options_setter_clearing(self):
+        config = self._get_target_class()()
+        config._properties["load"]["parquetOptions"] = dict(
+            enumAsString=False, enableListInference=True
+        )
+
+        config.parquet_options = None
+        self.assertNotIn("parquetOptions", config._properties["load"])
diff --git a/tests/unit/test_external_config.py b/tests/unit/test_external_config.py
index 648a8717..7178367e 100644
--- a/tests/unit/test_external_config.py
+++ b/tests/unit/test_external_config.py
@@ -425,6 +425,106 @@ class TestExternalConfig(unittest.TestCase):
 
         self.assertEqual(got_resource, exp_resource)
 
+    def test_parquet_options_getter(self):
+        from google.cloud.bigquery.format_options import ParquetOptions
+
+        parquet_options = ParquetOptions.from_api_repr(
+            {"enumAsString": True, "enableListInference": False}
+        )
+        ec = external_config.ExternalConfig(
+            external_config.ExternalSourceFormat.PARQUET
+        )
+
+        self.assertIsNone(ec.parquet_options.enum_as_string)
+        self.assertIsNone(ec.parquet_options.enable_list_inference)
+
+        ec._options = parquet_options
+
+        self.assertTrue(ec.parquet_options.enum_as_string)
+        self.assertFalse(ec.parquet_options.enable_list_inference)
+
+        self.assertIs(ec.parquet_options, ec.options)
+
+    def test_parquet_options_getter_non_parquet_format(self):
+        ec = external_config.ExternalConfig(external_config.ExternalSourceFormat.CSV)
+        self.assertIsNone(ec.parquet_options)
+
+    def test_parquet_options_setter(self):
+        from google.cloud.bigquery.format_options import ParquetOptions
+
+        parquet_options = ParquetOptions.from_api_repr(
+            {"enumAsString": False, "enableListInference": True}
+        )
+        ec = external_config.ExternalConfig(
+            external_config.ExternalSourceFormat.PARQUET
+        )
+
+        ec.parquet_options = parquet_options
+
+        # Setting Parquet options should be reflected in the generic options attribute.
+        self.assertFalse(ec.options.enum_as_string)
+        self.assertTrue(ec.options.enable_list_inference)
+
+    def test_parquet_options_setter_non_parquet_format(self):
+        from google.cloud.bigquery.format_options import ParquetOptions
+
+        parquet_options = ParquetOptions.from_api_repr(
+            {"enumAsString": False, "enableListInference": True}
+        )
+        ec = external_config.ExternalConfig(external_config.ExternalSourceFormat.CSV)
+
+        with self.assertRaisesRegex(TypeError, "Cannot set.*source format is CSV"):
+            ec.parquet_options = parquet_options
+
+    def test_from_api_repr_parquet(self):
+        from google.cloud.bigquery.format_options import ParquetOptions
+
+        resource = _copy_and_update(
+            self.BASE_RESOURCE,
+            {
+                "sourceFormat": "PARQUET",
+                "parquetOptions": {"enumAsString": True, "enableListInference": False},
+            },
+        )
+
+        ec = external_config.ExternalConfig.from_api_repr(resource)
+
+        self._verify_base(ec)
+        self.assertEqual(ec.source_format, external_config.ExternalSourceFormat.PARQUET)
+        self.assertIsInstance(ec.options, ParquetOptions)
+        self.assertTrue(ec.parquet_options.enum_as_string)
+        self.assertFalse(ec.parquet_options.enable_list_inference)
+
+        got_resource = ec.to_api_repr()
+
+        self.assertEqual(got_resource, resource)
+
+        del resource["parquetOptions"]["enableListInference"]
+        ec = external_config.ExternalConfig.from_api_repr(resource)
+        self.assertIsNone(ec.options.enable_list_inference)
+        got_resource = ec.to_api_repr()
+        self.assertEqual(got_resource, resource)
+
+    def test_to_api_repr_parquet(self):
+        from google.cloud.bigquery.format_options import ParquetOptions
+
+        ec = external_config.ExternalConfig(
+            external_config.ExternalSourceFormat.PARQUET
+        )
+        options = ParquetOptions.from_api_repr(
+            dict(enumAsString=False, enableListInference=True)
+        )
+        ec._options = options
+
+        exp_resource = {
+            "sourceFormat": external_config.ExternalSourceFormat.PARQUET,
+            "parquetOptions": {"enumAsString": False, "enableListInference": True},
+        }
+
+        got_resource = ec.to_api_repr()
+
+        self.assertEqual(got_resource, exp_resource)
+
 
 def _copy_and_update(d, u):
     d = copy.deepcopy(d)
diff --git a/tests/unit/test_format_options.py b/tests/unit/test_format_options.py
new file mode 100644
index 00000000..ab5f9e05
--- /dev/null
+++ b/tests/unit/test_format_options.py
@@ -0,0 +1,41 @@
+# Copyright 2021 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+class TestParquetOptions:
+    @staticmethod
+    def _get_target_class():
+        from google.cloud.bigquery.format_options import ParquetOptions
+
+        return ParquetOptions
+
+    def test_ctor(self):
+        config = self._get_target_class()()
+        assert config.enum_as_string is None
+        assert config.enable_list_inference is None
+
+    def test_from_api_repr(self):
+        config = self._get_target_class().from_api_repr(
+            {"enumAsString": False, "enableListInference": True}
+        )
+        assert not config.enum_as_string
+        assert config.enable_list_inference
+
+    def test_to_api_repr(self):
+        config = self._get_target_class()()
+        config.enum_as_string = True
+        config.enable_list_inference = False
+
+        result = config.to_api_repr()
+        assert result == {"enumAsString": True, "enableListInference": False}