Repository: googleapis/python-bigquery

Issue:
Python 3.10 support
Hello!

It would be great to get python 3.10 supported by this excellent package.

Thanks,
Rustam


Discussion:
Thanks for the request! We're working on making sure our test infrastructure and dependencies support Python 3.10 first.
Now that pandas (https://pypi.org/project/pandas/1.3.4/#files) and pyarrow (https://pypi.org/project/pyarrow/6.0.0/#files) have made releases that support Python 3.10, I believe this is unblocked.
Looks like we're missing a `geopandas` dependency named `fiona`, but since that's very much an optional dependency, we should be able to support Python 3.10 with some adjustments to our tests to omit `geopandas`.
Locally, there's one unit test failure I need to investigate. TBD re: system tests

Edit: Unit test failure went away once I installed the google-cloud-bigquery-storage package.

<details>
```
(dev-3.10) ➜  python-bigquery git:(issue1006-python310) pytest tests/unit
====================================================== test session starts ======================================================
platform darwin -- Python 3.10.0, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /Users/swast/src/github.com/googleapis/python-bigquery
collected 1889 items / 1 skipped / 1888 selected                                                                                

tests/unit/test__helpers.py sssssss...................................................................................... [  4%]
...................................................................                                                       [  8%]
tests/unit/test__http.py ......                                                                                           [  8%]
tests/unit/test__pandas_helpers.py ........................................................................sss........... [ 13%]
...............s...sssss........                                                                                          [ 15%]
tests/unit/test_client.py .........................................................................sss.s................. [ 20%]
......................................................................s.................................................. [ 26%]
....................                                                                                                      [ 27%]
tests/unit/test_create_dataset.py .............                                                                           [ 28%]
tests/unit/test_dataset.py ................................................................................               [ 32%]
tests/unit/test_dbapi__helpers.py ................................................................................        [ 36%]
tests/unit/test_dbapi_connection.py s..s.ss.s....                                                                         [ 37%]
tests/unit/test_dbapi_cursor.py ........................ssss...............................                               [ 40%]
tests/unit/test_dbapi_types.py ...........                                                                                [ 41%]
tests/unit/test_delete_dataset.py ...........                                                                             [ 41%]
tests/unit/test_encryption_configuration.py ..............                                                                [ 42%]
tests/unit/test_external_config.py ...........................................                                            [ 44%]
tests/unit/test_format_options.py ......                                                                                  [ 44%]
tests/unit/test_job_retry.py .........                                                                                    [ 45%]
tests/unit/test_list_datasets.py ....                                                                                     [ 45%]
tests/unit/test_list_jobs.py ........                                                                                     [ 46%]
tests/unit/test_list_models.py ..........                                                                                 [ 46%]
tests/unit/test_list_projects.py ....                                                                                     [ 46%]
tests/unit/test_list_routines.py ..........                                                                               [ 47%]
tests/unit/test_list_tables.py ..............                                                                             [ 48%]
tests/unit/test_opentelemetry_tracing.py sssssss                                                                          [ 48%]
tests/unit/test_query.py ................................................................................................ [ 53%]
...........................                                                                                               [ 55%]
tests/unit/test_retry.py ...................                                                                              [ 56%]
tests/unit/test_schema.py ............................................................................                    [ 60%]
tests/unit/test_signature_compatibility.py ..                                                                             [ 60%]
tests/unit/test_table.py ................................................................................................ [ 65%]
.....................s.......s.......s.Fssssss......s...s..ss......s..ssssssssssss..sss.ss............................... [ 71%]
............ssss                                                                                                          [ 72%]
tests/unit/enums/test_standard_sql_data_types.py ..s                                                                      [ 72%]
tests/unit/helpers/test_from_json.py ................................................                                     [ 75%]
tests/unit/job/test_base.py ...............................................................................               [ 79%]
tests/unit/job/test_copy.py ....................                                                                          [ 80%]
tests/unit/job/test_extract.py ................                                                                           [ 81%]
tests/unit/job/test_load.py .............................                                                                 [ 82%]
tests/unit/job/test_load_config.py ...................................................................................... [ 87%]
.......                                                                                                                   [ 87%]
tests/unit/job/test_query.py ...........................................................                                  [ 90%]
tests/unit/job/test_query_config.py ..........................                                                            [ 92%]
tests/unit/job/test_query_pandas.py ...........sssssss..sss..ss..sssssss                                                  [ 94%]
tests/unit/job/test_query_stats.py .........................                                                              [ 95%]
tests/unit/line_arg_parser/test_lexer.py .                                                                                [ 95%]
tests/unit/line_arg_parser/test_parser.py ...........                                                                     [ 96%]
tests/unit/line_arg_parser/test_visitors.py .                                                                             [ 96%]
tests/unit/model/test_model.py ....................                                                                       [ 97%]
tests/unit/model/test_model_reference.py ...........                                                                      [ 97%]
tests/unit/routine/test_routine.py .........................                                                              [ 99%]
tests/unit/routine/test_routine_argument.py ......                                                                        [ 99%]
tests/unit/routine/test_routine_reference.py ...........                                                                  [100%]

=========================================================== FAILURES ============================================================
______________________________ TestRowIterator.test_to_arrow_ensure_bqstorage_client_wo_bqstorage _______________________________

self = <tests.unit.test_table.TestRowIterator testMethod=test_to_arrow_ensure_bqstorage_client_wo_bqstorage>

    @unittest.skipIf(pyarrow is None, "Requires `pyarrow`")
    def test_to_arrow_ensure_bqstorage_client_wo_bqstorage(self):
        from google.cloud.bigquery.schema import SchemaField
    
        schema = [
            SchemaField("name", "STRING", mode="REQUIRED"),
            SchemaField("age", "INTEGER", mode="REQUIRED"),
        ]
        rows = [
            {"f": [{"v": "Alice"}, {"v": "98"}]},
            {"f": [{"v": "Bob"}, {"v": "99"}]},
        ]
        path = "/foo"
        api_request = mock.Mock(return_value={"rows": rows})
    
        mock_client = _mock_client()
        mock_client._ensure_bqstorage_client.return_value = None
        row_iterator = self._make_one(mock_client, api_request, path, schema)
    
        tbl = row_iterator.to_arrow(create_bqstorage_client=True)
    
        # The client attempted to create a BQ Storage client, and even though
        # that was not possible, results were still returned without errors.
>       mock_client._ensure_bqstorage_client.assert_called_once()

tests/unit/test_table.py:2570: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

_mock_self = <MagicMock name='mock._ensure_bqstorage_client' spec='function' id='4987555360'>

    def assert_called_once(_mock_self):
        """assert that the mock was called only once.
        """
        self = _mock_self
        if not self.call_count == 1:
            msg = ("Expected '%s' to have been called once. Called %s times.%s"
                   % (self._mock_name or 'mock',
                      self.call_count,
                      self._calls_repr()))
>           raise AssertionError(msg)
E           AssertionError: Expected '_ensure_bqstorage_client' to have been called once. Called 0 times.

/usr/local/Caskroom/miniconda/base/envs/dev-3.10/lib/python3.10/site-packages/mock/mock.py:891: AssertionError
==================================================== short test summary info ====================================================
FAILED tests/unit/test_table.py::TestRowIterator::test_to_arrow_ensure_bqstorage_client_wo_bqstorage - AssertionError: Expecte...
========================================== 1 failed, 1796 passed, 93 skipped in 30.84s ==========================================
```
</details>
System tests pass locally, as well.

```
(dev-3.10) ➜  python-bigquery git:(issue1006-python310) pytest tests/system                 
====================================================== test session starts ======================================================
platform darwin -- Python 3.10.0, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /Users/swast/src/github.com/googleapis/python-bigquery
collected 106 items                                                                                                             

tests/system/test_arrow.py ......                                                                                         [  5%]
tests/system/test_client.py .....................................................................                         [ 70%]
tests/system/test_job_retry.py ..                                                                                         [ 72%]
tests/system/test_list_rows.py ....                                                                                       [ 76%]
tests/system/test_magics.py .                                                                                             [ 77%]
tests/system/test_pandas.py ...............sssss                                                                          [ 96%]
tests/system/test_query.py ..                                                                                             [ 98%]
tests/system/test_structs.py ..                                                                                           [100%]

======================================================= warnings summary ========================================================
../python-bigquery-storage/google/cloud/bigquery_storage_v1/services/big_query_read/client.py:17
  /Users/swast/src/github.com/googleapis/python-bigquery-storage/google/cloud/bigquery_storage_v1/services/big_query_read/client.py:17: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
    from distutils import util

tests/system/test_magics.py::test_bigquery_magic
  /Users/swast/src/github.com/googleapis/python-bigquery/google/cloud/bigquery/job/query.py:1643: UserWarning: A progress bar was requested, but there was an error loading the tqdm library. Please install tqdm to use the progress bar functionality.
    query_result = wait_for_query(self, progress_bar_type, max_results=max_results)

tests/system/test_magics.py::test_bigquery_magic
  /Users/swast/src/github.com/googleapis/python-bigquery/google/cloud/bigquery/table.py:1938: UserWarning: A progress bar was requested, but there was an error loading the tqdm library. Please install tqdm to use the progress bar functionality.
    record_batch = self.to_arrow(

-- Docs: https://docs.pytest.org/en/stable/warnings.html
==================================== 101 passed, 5 skipped, 3 warnings in 702.45s (0:11:42) =====================================
```
Re-opening because I missed the need to exclude geopandas from the Python 3.10 snippets session. See failure here: https://source.cloud.google.com/results/invocations/f49f91e3-f6a0-48e9-b2d5-87410cc213b4/targets

Resolving patch:
diff --git a/noxfile.py b/noxfile.py
--- a/noxfile.py
+++ b/noxfile.py
@@ -186,7 +186,11 @@ def snippets(session):
     session.install("google-cloud-storage", "-c", constraints_path)
     session.install("grpcio", "-c", constraints_path)
 
-    session.install("-e", ".[all]", "-c", constraints_path)
+    if session.python == "3.10":
+        extras = "[bqstorage,pandas,tqdm,opentelemetry]"
+    else:
+        extras = "[all]"
+    session.install("-e", f".{extras}", "-c", constraints_path)
 
     # Run py.test against the snippets tests.
     # Skip tests in samples/snippets, as those are run in a different session


Test patch: