Repository: googleapis/python-bigquery

Issue:
Add "continuous" job for prerelease deps tests
Follow-up to https://github.com/googleapis/python-bigquery/issues/95

We should have a continuous (nightly) job that tests with presubmits. We usually have PRs open, but it'd be good to have timely notifications from flakey bot if something happens when we aren't making a PR.

Also, we should double-check that as we add pyarrow and such to required dependencies that we're still able to test with pre-release versions. It'd be good to ignore our pins so that we know when a dependency makes a breaking change.


Discussion:
I checked the logs for a recent run: https://source.cloud.google.com/results/invocations/c8fe14c7-35f5-4d8f-b1bc-7565a2c7240e/targets/cloud-devrel%2Fclient-libraries%2Fpython%2Fgoogleapis%2Fpython-bigquery%2Fpresubmit%2Fprerelease-deps-3.8/log

We haven't been using the nightly pre-release for Arrow or Pandas in the actual session.

```
nox > python -c import grpc; print(grpc.__version__)
1.40.0
nox > python -c import pandas; print(pandas.__version__)
1.3.2
nox > python -c import pyarrow; print(pyarrow.__version__)
5.0.0
```

I think this is because `python -m pip install -e .[all]` reinstalls them. Maybe there's an argument I can use to keep the already installed versions, even if we currently exclude them?

Resolving patch:
diff --git a/noxfile.py b/noxfile.py
--- a/noxfile.py
+++ b/noxfile.py
@@ -16,6 +16,7 @@
 
 import pathlib
 import os
+import re
 import shutil
 
 import nox
@@ -212,16 +213,30 @@ def prerelease_deps(session):
     # PyArrow prerelease packages are published to an alternative PyPI host.
     # https://arrow.apache.org/docs/python/install.html#installing-nightly-packages
     session.install(
-        "--extra-index-url", "https://pypi.fury.io/arrow-nightlies/", "--pre", "pyarrow"
+        "--extra-index-url",
+        "https://pypi.fury.io/arrow-nightlies/",
+        "--prefer-binary",
+        "--pre",
+        "--upgrade",
+        "pyarrow",
     )
     session.install(
+        "--extra-index-url",
+        "https://pypi.anaconda.org/scipy-wheels-nightly/simple",
+        "--prefer-binary",
         "--pre",
+        "--upgrade",
+        "pandas",
+    )
+
+    session.install(
+        "--pre",
+        "--upgrade",
         "google-api-core",
         "google-cloud-bigquery-storage",
         "google-cloud-core",
         "google-resumable-media",
         "grpcio",
-        "pandas",
     )
     session.install(
         "freezegun",
@@ -234,7 +249,30 @@ def prerelease_deps(session):
         "pytest",
         "pytest-cov",
     )
-    session.install("-e", ".[all]")
+
+    # Because we test minimum dependency versions on the minimum Python
+    # version, the first version we test with in the unit tests sessions has a
+    # constraints file containing all dependencies and extras.
+    with open(
+        CURRENT_DIRECTORY
+        / "testing"
+        / f"constraints-{UNIT_TEST_PYTHON_VERSIONS[0]}.txt",
+        encoding="utf-8",
+    ) as constraints_file:
+        constraints_text = constraints_file.read()
+
+    # Ignore leading whitespace and comment lines.
+    deps = [
+        match.group(1)
+        for match in re.finditer(
+            r"^\s*(\S+)(?===\S+)", constraints_text, flags=re.MULTILINE
+        )
+    ]
+
+    # We use --no-deps to ensure that pre-release versions aren't overwritten
+    # by the version ranges in setup.py.
+    session.install(*deps)
+    session.install("--no-deps", "-e", ".[all]")
 
     # Print out prerelease package versions.
     session.run("python", "-c", "import grpc; print(grpc.__version__)")


Test patch: