Repository: googleapis/python-bigquery

Issue:
standardize samples directory layout
We have an odd mix of sample locations:

- `docs/snippets.py` -- the original location, not very copy-pasteable / browsable
- lots of samples in `samples/*.py`, but the test drivers are in `samples/tests/`
- `samples/snippets/*.py`
- `samples/geography/*.py`

To align with the infrastructure in https://github.com/googleapis/synthtool/tree/master/synthtool/gcp/templates/python_samples, the goal is to have:

- `samples/snippets/` -- these samples just require the `google-cloud-bigquery` library 
- `samples/geography/` -- require extra dependencies for working with geography data
- `sample/pandas/` -- require extra dependencies for working with pandas dataframes
- more??? -- possibly a directory for the user-based authentication samples

To avoid breaking cloud.google.com docs, this needs to be done in several steps:

1. Copy samples to correct location
2. Update docs (Googler must do this)
3. Delete duplicate samples in old location

Towards: https://github.com/googleapis/python-bigquery/issues/366 (use templated noxfile) and ~https://github.com/googleapis/python-bigquery/issues/765 (blacken samples using templated noxfile(s))~


Discussion:
This is a comment from another open ticket. Consolidating here and closing that other ticket.

Ideally we'd have some code samples which only require the bare minimum dependencies (no pandas, no google-cloud-bigquery-storage, no protobuf, etc). The samples/snippets directory would be perfect for this, but:

There are a couple of pandas-related samples there which ideally should be moved to a samples/pandas directory.
Likewise, there is a "desktop application" sample https://github.com/googleapis/python-bigquery/blob/main/samples/snippets/user_credentials.py which ideally belongs in its own directory with its own noxfile.py and requirements.txt. This is because it's responsible for the google-auth-oauthlib dependency, which isn't required for any other samples. Maybe we call this directory samples/desktopapp or something?
Related issues:

There are still code samples that haven't been moved to subdirectories: https://github.com/googleapis/python-bigquery/issues/790

Resolving patch:
diff --git a/samples/desktopapp/conftest.py b/samples/desktopapp/conftest.py
new file mode 100644
--- /dev/null
+++ b/samples/desktopapp/conftest.py
@@ -0,0 +1,23 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from google.cloud import bigquery
+import pytest
+
+
+@pytest.fixture
+def bigquery_client_patch(
+    monkeypatch: pytest.MonkeyPatch, bigquery_client: bigquery.Client
+) -> None:
+    monkeypatch.setattr(bigquery, "Client", lambda: bigquery_client)
diff --git a/samples/notebooks/conftest.py b/samples/notebooks/conftest.py
new file mode 100644
--- /dev/null
+++ b/samples/notebooks/conftest.py
@@ -0,0 +1,23 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from google.cloud import bigquery
+import pytest
+
+
+@pytest.fixture
+def bigquery_client_patch(
+    monkeypatch: pytest.MonkeyPatch, bigquery_client: bigquery.Client
+) -> None:
+    monkeypatch.setattr(bigquery, "Client", lambda: bigquery_client)
diff --git a/samples/snippets/user_credentials.py b/samples/snippets/user_credentials.py
deleted file mode 100644
--- a/samples/snippets/user_credentials.py
+++ /dev/null
@@ -1,78 +0,0 @@
-#!/usr/bin/env python
-
-# Copyright 2017 Google Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-"""Command-line application to run a query using user credentials.
-
-You must supply a client secrets file, which would normally be bundled with
-your application.
-"""
-
-import argparse
-
-
-def main(project: str) -> None:
-    # [START bigquery_auth_user_flow]
-    from google_auth_oauthlib import flow
-
-    # A local server is used as the callback URL in the auth flow.
-    appflow = flow.InstalledAppFlow.from_client_secrets_file(
-        "client_secrets.json", scopes=["https://www.googleapis.com/auth/bigquery"]
-    )
-
-    # This launches a local server to be used as the callback URL in the desktop
-    # app auth flow. If you are accessing the application remotely, such as over
-    # SSH or a remote Jupyter notebook, this flow will not work. Use the
-    # `gcloud auth application-default login --no-browser` command or workload
-    # identity federation to get authentication tokens, instead.
-    #
-    appflow.run_local_server()
-
-    credentials = appflow.credentials
-    # [END bigquery_auth_user_flow]
-
-    # [START bigquery_auth_user_query]
-    from google.cloud import bigquery
-
-    # TODO: Uncomment the line below to set the `project` variable.
-    # project = 'user-project-id'
-    #
-    # The `project` variable defines the project to be billed for query
-    # processing. The user must have the bigquery.jobs.create permission on
-    # this project to run a query. See:
-    # https://cloud.google.com/bigquery/docs/access-control#permissions
-
-    client = bigquery.Client(project=project, credentials=credentials)
-
-    query_string = """SELECT name, SUM(number) as total
-    FROM `bigquery-public-data.usa_names.usa_1910_current`
-    WHERE name = 'William'
-    GROUP BY name;
-    """
-    query_job = client.query(query_string)
-
-    # Print the results.
-    for row in query_job.result():  # Wait for the job to complete.
-        print("{}: {}".format(row["name"], row["total"]))
-    # [END bigquery_auth_user_query]
-
-
-if __name__ == "__main__":
-    parser = argparse.ArgumentParser(
-        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
-    )
-    parser.add_argument("project", help="Project to use for BigQuery billing.")
-    args = parser.parse_args()
-    main(args.project)


Test patch:
diff --git a/samples/snippets/jupyter_tutorial_test.py b/samples/snippets/jupyter_tutorial_test.py
deleted file mode 100644
--- a/samples/snippets/jupyter_tutorial_test.py
+++ /dev/null
@@ -1,175 +0,0 @@
-# Copyright 2018 Google Inc. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import typing
-from typing import Iterator
-
-import pytest
-
-if typing.TYPE_CHECKING:
-    from IPython.terminal.interactiveshell import TerminalInteractiveShell
-
-IPython = pytest.importorskip("IPython")
-interactiveshell = pytest.importorskip("IPython.terminal.interactiveshell")
-tools = pytest.importorskip("IPython.testing.tools")
-matplotlib = pytest.importorskip("matplotlib")
-
-# Ignore semicolon lint warning because semicolons are used in notebooks
-# flake8: noqa E703
-
-
-@pytest.fixture(scope="session")
-def ipython() -> "TerminalInteractiveShell":
-    config = tools.default_config()
-    config.TerminalInteractiveShell.simple_prompt = True
-    shell = interactiveshell.TerminalInteractiveShell.instance(config=config)
-    return shell
-
-
-@pytest.fixture()
-def ipython_interactive(
-    request: pytest.FixtureRequest, ipython: "TerminalInteractiveShell"
-) -> Iterator["TerminalInteractiveShell"]:
-    """Activate IPython's builtin hooks
-
-    for the duration of the test scope.
-    """
-    with ipython.builtin_trap:
-        yield ipython
-
-
-def _strip_region_tags(sample_text: str) -> str:
-    """Remove blank lines and region tags from sample text"""
-    magic_lines = [
-        line for line in sample_text.split("\n") if len(line) > 0 and "# [" not in line
-    ]
-    return "\n".join(magic_lines)
-
-
-def test_jupyter_tutorial(ipython: "TerminalInteractiveShell") -> None:
-    matplotlib.use("agg")
-    ip = IPython.get_ipython()
-    ip.extension_manager.load_extension("google.cloud.bigquery")
-
-    sample = """
-    # [START bigquery_jupyter_magic_gender_by_year]
-    %%bigquery
-    SELECT
-        source_year AS year,
-        COUNT(is_male) AS birth_count
-    FROM `bigquery-public-data.samples.natality`
-    GROUP BY year
-    ORDER BY year DESC
-    LIMIT 15
-    # [END bigquery_jupyter_magic_gender_by_year]
-    """
-    result = ip.run_cell(_strip_region_tags(sample))
-    result.raise_error()  # Throws an exception if the cell failed.
-
-    sample = """
-    # [START bigquery_jupyter_magic_gender_by_year_var]
-    %%bigquery total_births
-    SELECT
-        source_year AS year,
-        COUNT(is_male) AS birth_count
-    FROM `bigquery-public-data.samples.natality`
-    GROUP BY year
-    ORDER BY year DESC
-    LIMIT 15
-    # [END bigquery_jupyter_magic_gender_by_year_var]
-    """
-    result = ip.run_cell(_strip_region_tags(sample))
-    result.raise_error()  # Throws an exception if the cell failed.
-
-    assert "total_births" in ip.user_ns  # verify that variable exists
-    total_births = ip.user_ns["total_births"]
-    # [START bigquery_jupyter_plot_births_by_year]
-    total_births.plot(kind="bar", x="year", y="birth_count")
-    # [END bigquery_jupyter_plot_births_by_year]
-
-    sample = """
-    # [START bigquery_jupyter_magic_gender_by_weekday]
-    %%bigquery births_by_weekday
-    SELECT
-        wday,
-        SUM(CASE WHEN is_male THEN 1 ELSE 0 END) AS male_births,
-        SUM(CASE WHEN is_male THEN 0 ELSE 1 END) AS female_births
-    FROM `bigquery-public-data.samples.natality`
-    WHERE wday IS NOT NULL
-    GROUP BY wday
-    ORDER BY wday ASC
-    # [END bigquery_jupyter_magic_gender_by_weekday]
-    """
-    result = ip.run_cell(_strip_region_tags(sample))
-    result.raise_error()  # Throws an exception if the cell failed.
-
-    assert "births_by_weekday" in ip.user_ns  # verify that variable exists
-    births_by_weekday = ip.user_ns["births_by_weekday"]
-    # [START bigquery_jupyter_plot_births_by_weekday]
-    births_by_weekday.plot(x="wday")
-    # [END bigquery_jupyter_plot_births_by_weekday]
-
-    # [START bigquery_jupyter_import_and_client]
-    from google.cloud import bigquery
-
-    client = bigquery.Client()
-    # [END bigquery_jupyter_import_and_client]
-
-    # [START bigquery_jupyter_query_plurality_by_year]
-    sql = """
-    SELECT
-        plurality,
-        COUNT(1) AS count,
-        year
-    FROM
-        `bigquery-public-data.samples.natality`
-    WHERE
-        NOT IS_NAN(plurality) AND plurality > 1
-    GROUP BY
-        plurality, year
-    ORDER BY
-        count DESC
-    """
-    df = client.query(sql).to_dataframe()
-    df.head()
-    # [END bigquery_jupyter_query_plurality_by_year]
-
-    # [START bigquery_jupyter_plot_plurality_by_year]
-    pivot_table = df.pivot(index="year", columns="plurality", values="count")
-    pivot_table.plot(kind="bar", stacked=True, figsize=(15, 7))
-    # [END bigquery_jupyter_plot_plurality_by_year]
-
-    # [START bigquery_jupyter_query_births_by_gestation]
-    sql = """
-    SELECT
-        gestation_weeks,
-        COUNT(1) AS count
-    FROM
-        `bigquery-public-data.samples.natality`
-    WHERE
-        NOT IS_NAN(gestation_weeks) AND gestation_weeks <> 99
-    GROUP BY
-        gestation_weeks
-    ORDER BY
-        gestation_weeks
-    """
-    df = client.query(sql).to_dataframe()
-    # [END bigquery_jupyter_query_births_by_gestation]
-
-    # [START bigquery_jupyter_plot_births_by_gestation]
-    ax = df.plot(kind="bar", x="gestation_weeks", y="count", figsize=(15, 7))
-    ax.set_title("Count of Births by Gestation Weeks")
-    ax.set_xlabel("Gestation Weeks")
-    ax.set_ylabel("Count")
-    # [END bigquery_jupyter_plot_births_by_gestation]
diff --git a/samples/snippets/user_credentials_test.py b/samples/snippets/user_credentials_test.py
deleted file mode 100644
--- a/samples/snippets/user_credentials_test.py
+++ /dev/null
@@ -1,45 +0,0 @@
-# Copyright 2017 Google Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-import os
-from typing import Iterator, Union
-
-import google.auth
-import mock
-import pytest
-
-from user_credentials import main  # type: ignore
-
-PROJECT = os.environ["GOOGLE_CLOUD_PROJECT"]
-
-MockType = Union[mock.mock.MagicMock, mock.mock.AsyncMock]
-
-
-@pytest.fixture
-def mock_flow() -> Iterator[MockType]:
-    flow_patch = mock.patch("google_auth_oauthlib.flow.InstalledAppFlow", autospec=True)
-
-    with flow_patch as flow_mock:
-        flow_mock.from_client_secrets_file.return_value = flow_mock
-        flow_mock.credentials = google.auth.default()[0]
-        yield flow_mock
-
-
-def test_auth_query_console(
-    mock_flow: MockType, capsys: pytest.CaptureFixture[str]
-) -> None:
-    main(PROJECT)
-    out, _ = capsys.readouterr()
-    # Fun fact: William P. Wood was the 1st director of the US Secret Service.
-    assert "William" in out