Repository: allenai/allennlp (language: python)

Issue:
Remove upper bounds for requirements
**Is your feature request related to a problem? Please describe.**

Remove upper bounds for dependencies in `requirements.txt`, where possible

**Describe the solution you'd like**

I would like the upper bounds for versions in `requirements.txt` to removed except for cases that are known to be broken. In #4324, the stated reason for this pinning was to prevent CI breakage. It also stated that you had a bot to upgrade dependencies automatically. However, now that the library is being retired and Depdendabot was disabled in 20df7cdd3eea7f895ceee9c57e2be1a843510748, this upgrade process is no longer happening.

I would like to keep using this library, at least for a little while, and removing these upper bounds would make that much easier. I feel that the circumstances mean the original reasons might not apply and that the topic should be reconsidered. To prevent CI breakage after upper bound removal, they can be kept in a [constraints file](https://pip.pypa.io/en/stable/user_guide/#constraints-files) for use in CI. It would be up to end users in the future to limit dependency conflicts if they encounter any.

I understand if you don't want to go through the effort for this, but I can create a PR if this idea is approved.

**Describe alternatives you've considered**

I've considered forking this library and applying a patch so I can upgrade `spacy`:

```diff
diff --git a/requirements.txt b/requirements.txt
index a9514301d70..c2bbe742460 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -7,7 +7,7 @@ cached-path>=1.1.3,<1.2.0
 fairscale==0.4.6
 jsonnet>=0.10.0 ; sys.platform != 'win32'
 nltk>=3.6.5
-spacy>=2.1.0,<3.4
+spacy>=2.1.0,<3.5
 numpy>=1.21.4
 tensorboardX>=1.2
 requests>=2.28
```

However, everyone would need to maintain their own forks, which seems inefficient. Doing this centrally would save work overall and allow the library to be more usable after its official retirement.


Discussion / hints:
Hey @aphedges, yes this seems like a good solution to me. I'd be happy to review a PR.

Fix patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index d920b933e51..a2d50e18e13 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -21,8 +21,8 @@ env:
   TORCH_VERSION: 1.12.0
   # TORCH_CPU_INSTALL: conda install pytorch torchvision torchaudio cpuonly -c pytorch
   # TORCH_GPU_INSTALL: conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
-  TORCH_CPU_INSTALL: pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cpu
-  TORCH_GPU_INSTALL: pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113
+  TORCH_CPU_INSTALL: pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cpu -c constraints.txt
+  TORCH_GPU_INSTALL: pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113 -c constraints.txt
   # Change this to invalidate existing cache.
   CACHE_PREFIX: v11
   # Disable tokenizers parallelism because this doesn't help, and can cause issues in distributed tests.
@@ -150,7 +150,7 @@ jobs:
       id: virtualenv-cache
       with:
         path: .venv
-        key: ${{ env.CACHE_PREFIX }}-${{ env.WEEK_NUMBER }}-${{ runner.os }}-${{ env.RUNNER_ARCH }}-${{ env.PYTHON_VERSION }}-${{ matrix.task.torch_platform }}-${{ hashFiles('setup.py') }}-${{ hashFiles('*requirements.txt') }}
+        key: ${{ env.CACHE_PREFIX }}-${{ env.WEEK_NUMBER }}-${{ runner.os }}-${{ env.RUNNER_ARCH }}-${{ env.PYTHON_VERSION }}-${{ matrix.task.torch_platform }}-${{ hashFiles('setup.py') }}-${{ hashFiles('*requirements.txt', 'constraints.txt') }}
 
     - name: Setup virtual environment (no cache hit)
       if: steps.virtualenv-cache.outputs.cache-hit != 'true'
@@ -276,7 +276,7 @@ jobs:
       id: virtualenv-cache
       with:
         path: .venv
-        key: ${{ env.CACHE_PREFIX }}-${{ env.WEEK_NUMBER }}-${{ runner.os }}-${{ env.RUNNER_ARCH }}-${{ env.PYTHON_VERSION }}-cpu-${{ hashFiles('setup.py') }}-${{ hashFiles('*requirements.txt') }}
+        key: ${{ env.CACHE_PREFIX }}-${{ env.WEEK_NUMBER }}-${{ runner.os }}-${{ env.RUNNER_ARCH }}-${{ env.PYTHON_VERSION }}-cpu-${{ hashFiles('setup.py') }}-${{ hashFiles('*requirements.txt', 'constraints.txt') }}
 
     - name: Setup virtual environment (no cache hit)
       if: steps.virtualenv-cache.outputs.cache-hit != 'true'
@@ -377,7 +377,7 @@ jobs:
 
     - name: Install core package
       run: |
-        pip install $(ls dist/*.whl)${{ matrix.flavor }}
+        pip install $(ls dist/*.whl)${{ matrix.flavor }} -c constraints.txt
 
     - name: Download NLTK prerequisites
       run: |
@@ -484,7 +484,7 @@ jobs:
       id: virtualenv-cache
       with:
         path: .venv
-        key: ${{ env.CACHE_PREFIX }}-${{ env.WEEK_NUMBER }}-${{ runner.os }}-${{ env.RUNNER_ARCH }}-${{ env.PYTHON_VERSION }}-cpu-${{ hashFiles('setup.py') }}-${{ hashFiles('*requirements.txt') }}
+        key: ${{ env.CACHE_PREFIX }}-${{ env.WEEK_NUMBER }}-${{ runner.os }}-${{ env.RUNNER_ARCH }}-${{ env.PYTHON_VERSION }}-cpu-${{ hashFiles('setup.py') }}-${{ hashFiles('*requirements.txt', 'constraints.txt') }}
 
     - name: Setup virtual environment (no cache hit)
       if: steps.virtualenv-cache.outputs.cache-hit != 'true'
diff --git a/Dockerfile b/Dockerfile
index 4fa5bef6e32..a182b2fba51 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -14,9 +14,10 @@ COPY allennlp/version.py allennlp/version.py
 COPY setup.py .
 COPY requirements.txt .
 COPY dev-requirements.txt .
+COPY constraints.txt .
 RUN touch allennlp/__init__.py \
     && touch README.md \
-    && pip install --no-cache-dir -e .[all]
+    && pip install --no-cache-dir -c constraints.txt -e .[all]
 
 # Now add the full package source and re-install just the package.
 COPY allennlp allennlp
diff --git a/Makefile b/Makefile
index b4d1788958e..ffddd1df4ac 100644
--- a/Makefile
+++ b/Makefile
@@ -10,7 +10,7 @@ MD_DOCS_CONF_SRC = mkdocs-skeleton.yml
 MD_DOCS_TGT = site/
 MD_DOCS_EXTRAS = $(addprefix $(MD_DOCS_ROOT),README.md CHANGELOG.md CONTRIBUTING.md)
 
-TORCH_INSTALL = pip install torch torchvision
+TORCH_INSTALL = pip install torch torchvision -c constraints.txt
 DOCKER_TORCH_VERSION = 1.12.0-cuda11.3-python3.8
 DOCKER_TEST_TORCH_VERSION = 1.12.0-cuda11.3-python3.8
 
@@ -115,7 +115,7 @@ install :
 	$(TORCH_INSTALL)
 	pip install --upgrade pip
 	pip install pip-tools
-	pip-compile requirements.txt dev-requirements.txt -o final_requirements.txt --allow-unsafe --rebuild --verbose
+	pip-compile requirements.in -o final_requirements.txt --allow-unsafe --rebuild --verbose
 	pip install -e . -r final_requirements.txt
 	# These nltk packages are used by the 'checklist' module.
 	$(NLTK_DOWNLOAD_CMD)
diff --git a/constraints.txt b/constraints.txt
new file mode 100644
index 00000000000..deafdad272b
--- /dev/null
+++ b/constraints.txt
@@ -0,0 +1,24 @@
+################################
+###### Core dependencies #######
+################################
+torch<1.13.0
+torchvision<0.14.0
+cached-path<1.2.0
+spacy<3.4
+transformers<4.21
+filelock<3.8
+wandb<0.13.0
+
+# Protobuf is a dependency of wandb and tensorboard, but they are missing this pin.
+protobuf<4.0.0
+
+# Required so pip-compile can properly resolve the pydantic version
+inflect<6.0
+
+##################################################
+###### Extra dependencies for integrations #######
+##################################################
+# NOTE: we use a special trailing comment on each line to denote which extras
+# each package is needed by. For example, checklist is needed by the 'checklist' extra
+# that you install with 'pip install allennlp[checklist]'.
+checklist==0.0.11  # needed by: checklist
diff --git a/requirements.in b/requirements.in
new file mode 100644
index 00000000000..17757c9b929
--- /dev/null
+++ b/requirements.in
@@ -0,0 +1,3 @@
+-c constraints.txt
+-r dev-requirements.txt
+-r requirements.txt
diff --git a/requirements.txt b/requirements.txt
index a9514301d70..07db152e4b9 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,13 +1,13 @@
 ################################
 ###### Core dependencies #######
 ################################
-torch>=1.10.0,<1.13.0
-torchvision>=0.8.1,<0.14.0
-cached-path>=1.1.3,<1.2.0
+torch>=1.10.0
+torchvision>=0.8.1
+cached-path>=1.1.3
 fairscale==0.4.6
 jsonnet>=0.10.0 ; sys.platform != 'win32'
 nltk>=3.6.5
-spacy>=2.1.0,<3.4
+spacy>=2.1.0
 numpy>=1.21.4
 tensorboardX>=1.2
 requests>=2.28
@@ -16,14 +16,14 @@ h5py>=3.6.0
 scikit-learn>=1.0.1
 scipy>=1.7.3
 pytest>=6.2.5
-transformers>=4.1,<4.21
+transformers>=4.1
 sentencepiece>=0.1.96
 dataclasses;python_version<'3.7'
-filelock>=3.3,<3.8
+filelock>=3.3
 lmdb>=1.2.1
 more-itertools>=8.12.0
 termcolor==1.1.0
-wandb>=0.10.0,<0.13.0
+wandb>=0.10.0
 huggingface_hub>=0.0.16
 dill>=0.3.4
 base58>=2.1.1
@@ -34,12 +34,8 @@ sacremoses
 # Spacy depends on typer, and typer had a bug. This is how we make sure we get the fixed version of typer.
 typer>=0.4.1
 
-# Indirect dependency of cached-path
-# pyasn1<0.5.0,>=0.4.8
-# pyasn1-modules>=0.2.8
-
 # Protobuf is a dependency of wandb and tensorboard, but they are missing this pin.
-protobuf>=3.12.0,<4.0.0
+protobuf>=3.12.0
 
 # We need this for building the Docker image
 traitlets>5.1.1
@@ -50,4 +46,4 @@ traitlets>5.1.1
 # NOTE: we use a special trailing comment on each line to denote which extras
 # each package is needed by. For example, checklist is needed by the 'checklist' extra
 # that you install with 'pip install allennlp[checklist]'.
-checklist==0.0.11  # needed by: checklist
+checklist>=0.0.11  # needed by: checklist
diff --git a/scripts/check_torch_version.py b/scripts/check_torch_version.py
index f6125c865ed..145046a1440 100644
--- a/scripts/check_torch_version.py
+++ b/scripts/check_torch_version.py
@@ -45,16 +45,16 @@ def _get_latest_torch_version() -> Tuple[str, str, str]:
 
 
 def _get_torch_version_upper_limit() -> Tuple[str, str, str]:
-    with open("requirements.txt") as f:
+    with open("constraints.txt") as f:
         for line in f:
             # The torch version line should look like:
-            #   "torch>=X.Y.Z,<X.V.0",
-            if "torch>=" in line:
+            #   "torch<X.V.0",
+            if "torch<" in line:
                 version = tuple(line.split("<")[1].strip().split("."))
                 assert len(version) == 3, f"Bad parsed version '{version}'"
                 break
         else:
-            raise RuntimeError("could not find torch version spec in requirements.txt")
+            raise RuntimeError("could not find torch version spec in constraints.txt")
     return cast(Tuple[str, str, str], version)
 
 


Test patch:
diff --git a/Dockerfile.test b/Dockerfile.test
index 42d592a8ede..e1981a9ef91 100644
--- a/Dockerfile.test
+++ b/Dockerfile.test
@@ -16,9 +16,10 @@ WORKDIR /stage/allennlp
 COPY allennlp/version.py allennlp/version.py
 COPY setup.py .
 COPY dev-requirements.txt .
+COPY constraints.txt .
 RUN touch allennlp/__init__.py \
     && touch README.md \
-    && pip install --no-cache-dir -e . -r dev-requirements.txt
+    && pip install --no-cache-dir -c constraints.txt -e . -r dev-requirements.txt
 
 # Now add the full package source and re-install just the package.
 COPY . .