Repository: vllm-project/vllm
Pull request #16859: Update PyTorch to 2.7.0
Author: login: huydhn
Labels: name: documentation
description: Improvements or additions to documentation
name: ci/build
Created: 2025-04-18T16:55:05Z | Merged: 2025-04-30T02:08:04Z

Notable changes:

* PyTorch 2.7.0 has dropped CUDA 12.4, so the remaining options are 12.6 and 12.8 
    * CUDA 12.6 has build issue https://github.com/vllm-project/vllm/issues/15435#issuecomment-2775924628, so only 12.8 remains
* We need a new ~~xformers~~ (0.0.30 is ready now), flashinfer, and mamba-ssm packages, so let build them from source for now.  They can be installed from pypi once they are built upstream with 2.7.0
* Leave XPU for later for Intel folks to pick it up as it requires a newer version of `intel-extension-for-pytorch`
* Update the release pipeline to build with CUDA 12.8 too


Diff:
diff --git a/.buildkite/release-pipeline.yaml b/.buildkite/release-pipeline.yaml
index a21a657c4b05..65f1fbd17e28 100644
--- a/.buildkite/release-pipeline.yaml
+++ b/.buildkite/release-pipeline.yaml
@@ -1,20 +1,20 @@
 steps:
-  - label: "Build wheel - CUDA 12.4"
+  - label: "Build wheel - CUDA 12.8"
     agents:
       queue: cpu_queue_postmerge
     commands:
-      - "DOCKER_BUILDKIT=1 docker build --build-arg max_jobs=16 --build-arg USE_SCCACHE=1 --build-arg GIT_REPO_CHECK=1 --build-arg CUDA_VERSION=12.4.0 --tag vllm-ci:build-image --target build --progress plain -f docker/Dockerfile ."
+      - "DOCKER_BUILDKIT=1 docker build --build-arg max_jobs=16 --build-arg USE_SCCACHE=1 --build-arg GIT_REPO_CHECK=1 --build-arg CUDA_VERSION=12.8.1 --tag vllm-ci:build-image --target build --progress plain -f docker/Dockerfile ."
       - "mkdir artifacts"
       - "docker run --rm -v $(pwd)/artifacts:/artifacts_host vllm-ci:build-image bash -c 'cp -r dist /artifacts_host && chmod -R a+rw /artifacts_host'"
       - "bash .buildkite/scripts/upload-wheels.sh"
     env:
       DOCKER_BUILDKIT: "1"
 
-  - label: "Build wheel - CUDA 12.1"
+  - label: "Build wheel - CUDA 12.6"
     agents:
       queue: cpu_queue_postmerge
     commands:
-      - "DOCKER_BUILDKIT=1 docker build --build-arg max_jobs=16 --build-arg USE_SCCACHE=1 --build-arg GIT_REPO_CHECK=1 --build-arg CUDA_VERSION=12.1.0 --tag vllm-ci:build-image --target build --progress plain -f docker/Dockerfile ."
+      - "DOCKER_BUILDKIT=1 docker build --build-arg max_jobs=16 --build-arg USE_SCCACHE=1 --build-arg GIT_REPO_CHECK=1 --build-arg CUDA_VERSION=12.6.3 --tag vllm-ci:build-image --target build --progress plain -f docker/Dockerfile ."
       - "mkdir artifacts"
       - "docker run --rm -v $(pwd)/artifacts:/artifacts_host vllm-ci:build-image bash -c 'cp -r dist /artifacts_host && chmod -R a+rw /artifacts_host'"
       - "bash .buildkite/scripts/upload-wheels.sh"
@@ -48,7 +48,7 @@ steps:
       queue: cpu_queue_postmerge
     commands:
       - "aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7"
-      - "DOCKER_BUILDKIT=1 docker build --build-arg max_jobs=16 --build-arg USE_SCCACHE=1 --build-arg GIT_REPO_CHECK=1 --build-arg CUDA_VERSION=12.4.0 --tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT --target vllm-openai --progress plain -f docker/Dockerfile ."
+      - "DOCKER_BUILDKIT=1 docker build --build-arg max_jobs=16 --build-arg USE_SCCACHE=1 --build-arg GIT_REPO_CHECK=1 --build-arg CUDA_VERSION=12.8.1 --tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT --target vllm-openai --progress plain -f docker/Dockerfile ."
       - "docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT"
 
   - label: "Build and publish TPU release image"
diff --git a/.buildkite/scripts/upload-wheels.sh b/.buildkite/scripts/upload-wheels.sh
index a681f8927060..75e3ef264095 100644
--- a/.buildkite/scripts/upload-wheels.sh
+++ b/.buildkite/scripts/upload-wheels.sh
@@ -50,11 +50,11 @@ aws s3 cp "$normal_wheel" "s3://vllm-wheels/$BUILDKITE_COMMIT/"
 if [[ $normal_wheel == *"cu118"* ]]; then
     # if $normal_wheel matches cu118, do not upload the index.html
     echo "Skipping index files for cu118 wheels"
-elif [[ $normal_wheel == *"cu121"* ]]; then
-    # if $normal_wheel matches cu121, do not upload the index.html
-    echo "Skipping index files for cu121 wheels"
+elif [[ $normal_wheel == *"cu126"* ]]; then
+    # if $normal_wheel matches cu126, do not upload the index.html
+    echo "Skipping index files for cu126 wheels"
 else
-    # only upload index.html for cu124 wheels (default wheels)
+    # only upload index.html for cu128 wheels (default wheels)
     aws s3 cp index.html "s3://vllm-wheels/$BUILDKITE_COMMIT/vllm/index.html"
     aws s3 cp "s3://vllm-wheels/nightly/index.html" "s3://vllm-wheels/$BUILDKITE_COMMIT/index.html"
 fi
@@ -66,12 +66,12 @@ aws s3 cp "$normal_wheel" "s3://vllm-wheels/nightly/"
 if [[ $normal_wheel == *"cu118"* ]]; then
     # if $normal_wheel matches cu118, do not upload the index.html
     echo "Skipping index files for cu118 wheels"
-elif [[ $normal_wheel == *"cu121"* ]]; then
-    # if $normal_wheel matches cu121, do not upload the index.html
-    echo "Skipping index files for cu121 wheels"
+elif [[ $normal_wheel == *"cu126"* ]]; then
+    # if $normal_wheel matches cu126, do not upload the index.html
+    echo "Skipping index files for cu126 wheels"
 else
-    # only upload index.html for cu124 wheels (default wheels)
+    # only upload index.html for cu128 wheels (default wheels)
     aws s3 cp index.html "s3://vllm-wheels/nightly/vllm/index.html"
 fi
 
-aws s3 cp "$wheel" "s3://vllm-wheels/$version/"
\ No newline at end of file
+aws s3 cp "$wheel" "s3://vllm-wheels/$version/"
diff --git a/.buildkite/test-pipeline.yaml b/.buildkite/test-pipeline.yaml
index 20d858cb15a1..b1204b52fab0 100644
--- a/.buildkite/test-pipeline.yaml
+++ b/.buildkite/test-pipeline.yaml
@@ -450,7 +450,7 @@ steps:
   - tests/models/encoder_decoder/language
   commands:
     # Install causal-conv1d for plamo2 models here, as it is not compatible with pip-compile.
-    - pip install causal-conv1d
+    - pip install 'git+https://github.com/Dao-AILab/causal-conv1d@v1.5.0.post8'
     - pytest -v -s models/decoder_only/language -m 'core_model or quant_model'
     - pytest -v -s models/embedding/language -m core_model
 
diff --git a/.github/workflows/lint-and-deploy.yaml b/.github/workflows/lint-and-deploy.yaml
index 7b1d9f69938c..dd9b61a64734 100644
--- a/.github/workflows/lint-and-deploy.yaml
+++ b/.github/workflows/lint-and-deploy.yaml
@@ -66,7 +66,7 @@ jobs:
           export AWS_SECRET_ACCESS_KEY=minioadmin
           sleep 30 && kubectl -n ns-vllm logs -f "$(kubectl -n ns-vllm get pods | awk '/deployment/ {print $1;exit}')" &
           helm install --wait --wait-for-jobs --timeout 5m0s --debug --create-namespace --namespace=ns-vllm test-vllm examples/online_serving/chart-helm -f examples/online_serving/chart-helm/values.yaml --set secrets.s3endpoint=http://minio:9000 --set secrets.s3bucketname=testbucket --set secrets.s3accesskeyid=$AWS_ACCESS_KEY_ID --set secrets.s3accesskey=$AWS_SECRET_ACCESS_KEY --set resources.requests.cpu=1 --set resources.requests.memory=4Gi --set resources.limits.cpu=2 --set resources.limits.memory=5Gi --set image.env[0].name=VLLM_CPU_KVCACHE_SPACE --set image.env[1].name=VLLM_LOGGING_LEVEL --set-string image.env[0].value="1" --set-string image.env[1].value="DEBUG" --set-string extraInit.s3modelpath="opt-125m/" --set-string 'resources.limits.nvidia\.com/gpu=0' --set-string 'resources.requests.nvidia\.com/gpu=0' --set-string image.repository="vllm-cpu-env"
-    
+
       - name: curl test
         run: |
           kubectl -n ns-vllm port-forward service/test-vllm-service 8001:80 &
@@ -79,4 +79,4 @@ jobs:
                           "max_tokens": 7,
                           "temperature": 0
                   }'):$CODE"
-          echo "$CODE"
\ No newline at end of file
+          echo "$CODE"
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index f76b24c025ff..148ef26d4335 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -46,7 +46,7 @@ repos:
   rev: 0.6.2
   hooks:
     - id: pip-compile
-      args: [requirements/test.in, -o, requirements/test.txt]
+      args: [requirements/test.in, -o, requirements/test.txt, --index-strategy, unsafe-best-match]
       files: ^requirements/test\.(in|txt)$
 - repo: local
   hooks:
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 3314f05fd2a0..6be9adcb8252 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -46,8 +46,8 @@ set(HIP_SUPPORTED_ARCHS "gfx906;gfx908;gfx90a;gfx942;gfx950;gfx1030;gfx1100;gfx1
 # requirements.txt files and should be kept consistent.  The ROCm torch
 # versions are derived from docker/Dockerfile.rocm
 #
-set(TORCH_SUPPORTED_VERSION_CUDA "2.6.0")
-set(TORCH_SUPPORTED_VERSION_ROCM "2.6.0")
+set(TORCH_SUPPORTED_VERSION_CUDA "2.7.0")
+set(TORCH_SUPPORTED_VERSION_ROCM "2.7.0")
 
 #
 # Try to find python package with an executable that exactly matches
diff --git a/docker/Dockerfile b/docker/Dockerfile
index 1b28845d0ac0..9605f5c76bb7 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -5,11 +5,11 @@
 # docs/source/contributing/dockerfile/dockerfile.md and
 # docs/source/assets/contributing/dockerfile-stages-dependency.png
 
-ARG CUDA_VERSION=12.4.1
+ARG CUDA_VERSION=12.8.1
 #################### BASE BUILD IMAGE ####################
 # prepare basic build environment
 FROM nvidia/cuda:${CUDA_VERSION}-devel-ubuntu20.04 AS base
-ARG CUDA_VERSION=12.4.1
+ARG CUDA_VERSION=12.8.1
 ARG PYTHON_VERSION=3.12
 ARG TARGETPLATFORM
 ENV DEBIAN_FRONTEND=noninteractive
@@ -34,6 +34,7 @@ RUN --mount=type=cache,target=/root/.cache/uv \
 # This timeout (in seconds) is necessary when installing some dependencies via uv since it's likely to time out
 # Reference: https://github.com/astral-sh/uv/pull/1694
 ENV UV_HTTP_TIMEOUT=500
+ENV UV_INDEX_STRATEGY="unsafe-best-match"
 
 # Upgrade to GCC 10 to avoid https://gcc.gnu.org/bugzilla/show_bug.cgi?id=92519
 # as it was causing spam when compiling the CUTLASS kernels
@@ -66,7 +67,8 @@ RUN --mount=type=cache,target=/root/.cache/uv \
 COPY requirements/common.txt requirements/common.txt
 COPY requirements/cuda.txt requirements/cuda.txt
 RUN --mount=type=cache,target=/root/.cache/uv \
-    uv pip install --system -r requirements/cuda.txt
+    uv pip install --system -r requirements/cuda.txt \
+    --extra-index-url https://download.pytorch.org/whl/cu$(echo $CUDA_VERSION | cut -d. -f1,2 | tr -d '.')
 
 # cuda arch list used by torch
 # can be useful for both `dev` and `test`
@@ -89,9 +91,11 @@ COPY requirements/build.txt requirements/build.txt
 # This timeout (in seconds) is necessary when installing some dependencies via uv since it's likely to time out
 # Reference: https://github.com/astral-sh/uv/pull/1694
 ENV UV_HTTP_TIMEOUT=500
+ENV UV_INDEX_STRATEGY="unsafe-best-match"
 
 RUN --mount=type=cache,target=/root/.cache/uv \
-    uv pip install --system -r requirements/build.txt
+    uv pip install --system -r requirements/build.txt \
+    --extra-index-url https://download.pytorch.org/whl/cu$(echo $CUDA_VERSION | cut -d. -f1,2 | tr -d '.')
 
 COPY . .
 ARG GIT_REPO_CHECK=0
@@ -158,22 +162,25 @@ FROM base as dev
 # This timeout (in seconds) is necessary when installing some dependencies via uv since it's likely to time out
 # Reference: https://github.com/astral-sh/uv/pull/1694
 ENV UV_HTTP_TIMEOUT=500
+ENV UV_INDEX_STRATEGY="unsafe-best-match"
+
+# Workaround for #17068
+RUN --mount=type=cache,target=/root/.cache/uv \
+    uv pip install --system --no-build-isolation "git+https://github.com/state-spaces/mamba@v2.2.4"
 
 COPY requirements/lint.txt requirements/lint.txt
 COPY requirements/test.txt requirements/test.txt
 COPY requirements/dev.txt requirements/dev.txt
-# Workaround for #17068
-RUN --mount=type=cache,target=/root/.cache/uv \
-    uv pip install --system mamba-ssm==2.2.4 --no-build-isolation
 RUN --mount=type=cache,target=/root/.cache/uv \
-    uv pip install --system -r requirements/dev.txt
+    uv pip install --system -r requirements/dev.txt \
+    --extra-index-url https://download.pytorch.org/whl/cu$(echo $CUDA_VERSION | cut -d. -f1,2 | tr -d '.')
 #################### DEV IMAGE ####################
 
 #################### vLLM installation IMAGE ####################
 # image with vLLM installed
 # TODO: Restore to base image after FlashInfer AOT wheel fixed
 FROM nvidia/cuda:${CUDA_VERSION}-devel-ubuntu22.04 AS vllm-base
-ARG CUDA_VERSION=12.4.1
+ARG CUDA_VERSION=12.8.1
 ARG PYTHON_VERSION=3.12
 WORKDIR /vllm-workspace
 ENV DEBIAN_FRONTEND=noninteractive
@@ -203,6 +210,7 @@ RUN --mount=type=cache,target=/root/.cache/uv \
 # This timeout (in seconds) is necessary when installing some dependencies via uv since it's likely to time out
 # Reference: https://github.com/astral-sh/uv/pull/1694
 ENV UV_HTTP_TIMEOUT=500
+ENV UV_INDEX_STRATEGY="unsafe-best-match"
 
 # Workaround for https://github.com/openai/triton/issues/2507 and
 # https://github.com/pytorch/pytorch/issues/107960 -- hopefully
@@ -223,7 +231,8 @@ RUN --mount=type=cache,target=/root/.cache/uv \
 # Install vllm wheel first, so that torch etc will be installed.
 RUN --mount=type=bind,from=build,src=/workspace/dist,target=/vllm-workspace/dist \
     --mount=type=cache,target=/root/.cache/uv \
-    uv pip install --system dist/*.whl --verbose
+    uv pip install --system dist/*.whl --verbose \
+    --extra-index-url https://download.pytorch.org/whl/cu$(echo $CUDA_VERSION | cut -d. -f1,2 | tr -d '.')
 
 # If we need to build FlashInfer wheel before its release:
 # $ export FLASHINFER_ENABLE_AOT=1
@@ -240,19 +249,26 @@ RUN --mount=type=bind,from=build,src=/workspace/dist,target=/vllm-workspace/dist
 RUN --mount=type=cache,target=/root/.cache/uv \
 . /etc/environment && \
 if [ "$TARGETPLATFORM" != "linux/arm64" ]; then \
-    uv pip install --system https://github.com/flashinfer-ai/flashinfer/releases/download/v0.2.1.post2/flashinfer_python-0.2.1.post2+cu124torch2.6-cp38-abi3-linux_x86_64.whl ; \
+    # TESTING: install FlashInfer from source to test 2.7.0 final RC
+    FLASHINFER_ENABLE_AOT=1 TORCH_CUDA_ARCH_LIST='7.5 8.0 8.6 8.9 9.0+PTX' \
+    uv pip install --system --no-build-isolation "git+https://github.com/flashinfer-ai/flashinfer@v0.2.2.post1" ; \
 fi
 COPY examples examples
 COPY benchmarks benchmarks
 COPY ./vllm/collect_env.py .
 
+RUN --mount=type=cache,target=/root/.cache/uv \
+. /etc/environment && \
+uv pip list
+
 # Although we build Flashinfer with AOT mode, there's still
 # some issues w.r.t. JIT compilation. Therefore we need to
 # install build dependencies for JIT compilation.
 # TODO: Remove this once FlashInfer AOT wheel is fixed
 COPY requirements/build.txt requirements/build.txt
 RUN --mount=type=cache,target=/root/.cache/uv \
-    uv pip install --system -r requirements/build.txt
+    uv pip install --system -r requirements/build.txt \
+    --extra-index-url https://download.pytorch.org/whl/cu$(echo $CUDA_VERSION | cut -d. -f1,2 | tr -d '.')
 
 #################### vLLM installation IMAGE ####################
 
@@ -266,11 +282,13 @@ ADD . /vllm-workspace/
 # This timeout (in seconds) is necessary when installing some dependencies via uv since it's likely to time out
 # Reference: https://github.com/astral-sh/uv/pull/1694
 ENV UV_HTTP_TIMEOUT=500
+ENV UV_INDEX_STRATEGY="unsafe-best-match"
 
-# install development dependencies (for testing)
 # Workaround for #17068
 RUN --mount=type=cache,target=/root/.cache/uv \
-    uv pip install --system mamba-ssm==2.2.4 --no-build-isolation
+    uv pip install --system --no-build-isolation "git+https://github.com/state-spaces/mamba@v2.2.4"
+
+# install development dependencies (for testing)
 RUN --mount=type=cache,target=/root/.cache/uv \
     uv pip install --system -r requirements/dev.txt
 
diff --git a/docs/source/getting_started/installation/gpu/cuda.inc.md b/docs/source/getting_started/installation/gpu/cuda.inc.md
index 46bdb08ebb77..06915f09dd51 100644
--- a/docs/source/getting_started/installation/gpu/cuda.inc.md
+++ b/docs/source/getting_started/installation/gpu/cuda.inc.md
@@ -1,6 +1,6 @@
 # Installation
 
-vLLM contains pre-compiled C++ and CUDA (12.1) binaries.
+vLLM contains pre-compiled C++ and CUDA (12.6) binaries.
 
 ## Requirements
 
@@ -23,12 +23,12 @@ Therefore, it is recommended to install vLLM with a **fresh new** environment. I
 You can install vLLM using either `pip` or `uv pip`:
 
 ```console
-# Install vLLM with CUDA 12.4.
+# Install vLLM with CUDA 12.6.
 pip install vllm # If you are using pip.
 uv pip install vllm # If you are using uv.
 ```
 
-As of now, vLLM's binaries are compiled with CUDA 12.4 and public PyTorch release versions by default. We also provide vLLM binaries compiled with CUDA 12.1, 11.8, and public PyTorch release versions:
+As of now, vLLM's binaries are compiled with CUDA 12.6 and public PyTorch release versions by default. We also provide vLLM binaries compiled with CUDA 12.8, 11.8, and public PyTorch release versions:
 
 ```console
 # Install vLLM with CUDA 11.8.
diff --git a/examples/online_serving/chart-helm/values.yaml b/examples/online_serving/chart-helm/values.yaml
index 9c48e7d061bf..28dba9a6f688 100644
--- a/examples/online_serving/chart-helm/values.yaml
+++ b/examples/online_serving/chart-helm/values.yaml
@@ -8,7 +8,7 @@ image:
   # -- Image tag
   tag: "latest"
   # -- Container launch command
-  command: ["vllm", "serve", "/data/", "--served-model-name", "opt-125m", "--dtype", "bfloat16", "--host", "0.0.0.0", "--port", "8000"]
+  command: ["vllm", "serve", "/data/", "--served-model-name", "opt-125m", "--dtype", "float32", "--block-size", "16", "--host", "0.0.0.0", "--port", "8000"]
 
 # -- Container port
 containerPort: 8000
diff --git a/pyproject.toml b/pyproject.toml
index b5f1039b44da..c85e85b0c82b 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -6,7 +6,7 @@ requires = [
     "packaging",
     "setuptools>=61",
     "setuptools-scm>=8.0",
-    "torch == 2.6.0",
+    "torch == 2.7.0",
     "wheel",
     "jinja2",
 ]
diff --git a/requirements/build.txt b/requirements/build.txt
index 13d643bcaff1..19d757b452ac 100644
--- a/requirements/build.txt
+++ b/requirements/build.txt
@@ -4,6 +4,6 @@ ninja
 packaging
 setuptools>=61
 setuptools-scm>=8
-torch==2.6.0
+torch==2.7.0
 wheel
 jinja2>=3.1.6
diff --git a/requirements/cpu.txt b/requirements/cpu.txt
index 69f732c2417a..752931158a05 100644
--- a/requirements/cpu.txt
+++ b/requirements/cpu.txt
@@ -2,18 +2,19 @@
 -r common.txt
 
 # Dependencies for CPUs
-torch==2.6.0+cpu; platform_machine == "x86_64"
-torch==2.6.0; platform_system == "Darwin"
-torch==2.6.0; platform_machine == "ppc64le" or platform_machine == "aarch64"
+--extra-index-url https://download.pytorch.org/whl/cpu
+torch==2.7.0+cpu; platform_machine == "x86_64"
+torch==2.7.0; platform_system == "Darwin"
+torch==2.7.0; platform_machine == "ppc64le" or platform_machine == "aarch64"
 torch==2.7.0.dev20250304; platform_machine == "s390x"
 
 # required for the image processor of minicpm-o-2_6, this must be updated alongside torch
 torchaudio; platform_machine != "ppc64le" and platform_machine != "s390x"
-torchaudio==2.6.0; platform_machine == "ppc64le"
+torchaudio==2.7.0; platform_machine == "ppc64le"
 
 # required for the image processor of phi3v, this must be updated alongside torch
 torchvision; platform_machine != "ppc64le" and platform_machine != "s390x"
-torchvision==0.21.0; platform_machine == "ppc64le"
+torchvision==0.22.0; platform_machine == "ppc64le"
 datasets # for benchmark scripts
 
 # cpu cannot use triton 3.3.0
diff --git a/requirements/cuda.txt b/requirements/cuda.txt
index cdc6ee75afbc..a71d9728f38a 100644
--- a/requirements/cuda.txt
+++ b/requirements/cuda.txt
@@ -6,8 +6,9 @@ numba == 0.61.2; python_version > '3.9'
 
 # Dependencies for NVIDIA GPUs
 ray[cgraph]>=2.43.0, !=2.44.* # Ray Compiled Graph, required for pipeline parallelism in V1.
-torch==2.6.0
-torchaudio==2.6.0
+torch==2.7.0
+torchaudio==2.7.0
 # These must be updated alongside torch
-torchvision==0.21.0 # Required for phi3v processor. See https://github.com/pytorch/vision?tab=readme-ov-file#installation for corresponding version
-xformers==0.0.29.post2; platform_system == 'Linux' and platform_machine == 'x86_64'  # Requires PyTorch 2.6.0
+torchvision==0.22.0 # Required for phi3v processor. See https://github.com/pytorch/vision?tab=readme-ov-file#installation for corresponding version
+# https://github.com/facebookresearch/xformers/releases/tag/v0.0.30
+xformers==0.0.30; platform_system == 'Linux' and platform_machine == 'x86_64'  # Requires PyTorch >= 2.7
diff --git a/requirements/rocm-build.txt b/requirements/rocm-build.txt
index 05de4ff16845..55ffe82e8177 100644
--- a/requirements/rocm-build.txt
+++ b/requirements/rocm-build.txt
@@ -2,9 +2,9 @@
 -r common.txt
 
 --extra-index-url https://download.pytorch.org/whl/rocm6.2.4
-torch==2.6.0
-torchvision==0.21.0
-torchaudio==2.6.0
+torch==2.7.0
+torchvision==0.22.0
+torchaudio==2.7.0
 
 triton==3.2
 cmake>=3.26,<4
diff --git a/requirements/test.in b/requirements/test.in
index c5d2c4cd4c30..ee79aae58d7e 100644
--- a/requirements/test.in
+++ b/requirements/test.in
@@ -23,9 +23,9 @@ sentence-transformers # required for embedding tests
 soundfile # required for audio tests
 jiwer # required for audio tests
 timm # required for internvl test
-torch==2.6.0
-torchaudio==2.6.0
-torchvision==0.21.0
+torch==2.7.0
+torchaudio==2.7.0
+torchvision==0.22.0
 transformers_stream_generator # required for qwen-vl test
 mamba_ssm # required for plamo2 test
 matplotlib # required for qwen-vl test
diff --git a/requirements/test.txt b/requirements/test.txt
index 9642a5bfe68d..2e8121e3882e 100644
--- a/requirements/test.txt
+++ b/requirements/test.txt
@@ -1,5 +1,5 @@
 # This file was autogenerated by uv via the following command:
-#    uv pip compile requirements/test.in -o requirements/test.txt
+#    uv pip compile requirements/test.in -o requirements/test.txt --index-strategy unsafe-best-match
 absl-py==2.1.0
     # via rouge-score
 accelerate==1.0.1
@@ -274,7 +274,7 @@ mamba-ssm==2.2.4
     # via -r requirements/test.in
 markdown-it-py==3.0.0
     # via rich
-markupsafe==3.0.2
+markupsafe==3.0.1
     # via
     #   jinja2
     #   werkzeug
@@ -355,39 +355,42 @@ numpy==1.26.4
     #   transformers
     #   tritonclient
     #   vocos
-nvidia-cublas-cu12==12.4.5.8
+nvidia-cublas-cu12==12.6.4.1
     # via
     #   nvidia-cudnn-cu12
     #   nvidia-cusolver-cu12
     #   torch
-nvidia-cuda-cupti-cu12==12.4.127
+nvidia-cuda-cupti-cu12==12.6.80
     # via torch
-nvidia-cuda-nvrtc-cu12==12.4.127
+nvidia-cuda-nvrtc-cu12==12.6.77
     # via torch
-nvidia-cuda-runtime-cu12==12.4.127
+nvidia-cuda-runtime-cu12==12.6.77
     # via torch
-nvidia-cudnn-cu12==9.1.0.70
+nvidia-cudnn-cu12==9.5.1.17
     # via torch
-nvidia-cufft-cu12==11.2.1.3
+nvidia-cufft-cu12==11.3.0.4
     # via torch
-nvidia-curand-cu12==10.3.5.147
+nvidia-cufile-cu12==1.11.1.6
     # via torch
-nvidia-cusolver-cu12==11.6.1.9
+nvidia-curand-cu12==10.3.7.77
     # via torch
-nvidia-cusparse-cu12==12.3.1.170
+nvidia-cusolver-cu12==11.7.1.2
+    # via torch
+nvidia-cusparse-cu12==12.5.4.2
     # via
     #   nvidia-cusolver-cu12
     #   torch
-nvidia-cusparselt-cu12==0.6.2
+nvidia-cusparselt-cu12==0.6.3
     # via torch
-nvidia-nccl-cu12==2.21.5
+nvidia-nccl-cu12==2.26.2
     # via torch
-nvidia-nvjitlink-cu12==12.4.127
+nvidia-nvjitlink-cu12==12.6.85
     # via
+    #   nvidia-cufft-cu12
     #   nvidia-cusolver-cu12
     #   nvidia-cusparse-cu12
     #   torch
-nvidia-nvtx-cu12==12.4.127
+nvidia-nvtx-cu12==12.6.77
     # via torch
 opencv-python-headless==4.11.0.86
     # via
@@ -634,6 +637,7 @@ setuptools==75.8.0
     #   mamba-ssm
     #   pytablewriter
     #   torch
+    #   triton
 shellingham==1.5.4
     # via typer
 six==1.16.0
@@ -664,7 +668,7 @@ starlette-testclient==0.4.1
     # via schemathesis
 statsmodels==0.14.4
     # via genai-perf
-sympy==1.13.1
+sympy==1.13.3
     # via
     #   einx
     #   torch
@@ -696,7 +700,7 @@ tomli==2.2.1
     # via schemathesis
 tomli-w==1.2.0
     # via schemathesis
-torch==2.6.0
+torch==2.7.0
     # via
     #   -r requirements/test.in
     #   accelerate
@@ -714,12 +718,12 @@ torch==2.6.0
     #   torchvision
     #   vector-quantize-pytorch
     #   vocos
-torchaudio==2.6.0
+torchaudio==2.7.0
     # via
     #   -r requirements/test.in
     #   encodec
     #   vocos
-torchvision==0.21.0
+torchvision==0.22.0
     # via
     #   -r requirements/test.in
     #   timm
@@ -748,7 +752,7 @@ transformers==4.51.3
     #   transformers-stream-generator
 transformers-stream-generator==0.0.5
     # via -r requirements/test.in
-triton==3.2.0
+triton==3.3.0
     # via torch
 tritonclient==2.51.0
     # via
diff --git a/setup.py b/setup.py
index a1867960e593..7675fbdf3efe 100755
--- a/setup.py
+++ b/setup.py
@@ -54,7 +54,7 @@ def load_module_from_path(module_name, path):
     # fallback to cpu
     VLLM_TARGET_DEVICE = "cpu"
 
-MAIN_CUDA_VERSION = "12.4"
+MAIN_CUDA_VERSION = "12.8"
 
 
 def is_sccache_available() -> bool:
diff --git a/vllm/attention/ops/ipex_attn.py b/vllm/attention/ops/ipex_attn.py
index 6d96f58320c8..1702203b1834 100644
--- a/vllm/attention/ops/ipex_attn.py
+++ b/vllm/attention/ops/ipex_attn.py
@@ -5,7 +5,8 @@
 try:
     import intel_extension_for_pytorch.llm.modules as ipex_modules
     _use_ipex = True
-except ImportError:
+# AttributeError is to handle a bug in ipex https://github.com/intel/intel-extension-for-pytorch/pull/813
+except (ImportError, AttributeError):
     _use_ipex = False
 
 import torch


Reviews:
user: login: drisspg
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-23T17:27:33Z
user: login: drisspg
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-23T17:28:07Z
user: login: huydhn
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-25T10:11:03Z
user: login: simon-mo
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-26T00:30:45Z
user: login: huydhn
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-26T00:52:21Z
user: login: huydhn
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-26T02:00:38Z
user: login: huydhn
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-26T04:09:40Z
user: login: mgoin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-26T15:39:18Z
user: login: huydhn
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-26T18:30:35Z
user: login: huydhn
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-26T18:31:51Z
user: login: cliffwoolley
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-30T01:00:26Z
user: login: huydhn
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T01:05:09Z
user: login: simon-mo
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-30T02:07:56Z

Review comments:
diff_hunk: @@ -77,6 +77,11 @@ ENV TORCH_CUDA_ARCH_LIST=${torch_cuda_arch_list}
 # Override the arch list for flash-attn to reduce the binary size
 ARG vllm_fa_cmake_gpu_arches='80-real;90-real'
 ENV VLLM_FA_CMAKE_GPU_ARCHES=${vllm_fa_cmake_gpu_arches}
+
+# TESTING: install xformers from source to test 2.7.0 final RC
+RUN --mount=type=cache,target=/root/.cache/uv \
+    TORCH_CUDA_ARCH_LIST='7.5 8.0 8.6 8.9 9.0+PTX' \
path: docker/Dockerfile
user: login: drisspg
body: for blackwell support we will wnat to add sm100 here, although the +PTX should handle this..
created_at: 2025-04-23T17:27:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -220,7 +225,14 @@ RUN --mount=type=cache,target=/root/.cache/uv \
 # Install vllm wheel first, so that torch etc will be installed.
 RUN --mount=type=bind,from=build,src=/workspace/dist,target=/vllm-workspace/dist \
     --mount=type=cache,target=/root/.cache/uv \
-    uv pip install --system dist/*.whl --verbose
+    uv pip install --system dist/*.whl --verbose \
+    --extra-index-url https://download.pytorch.org/whl/test/cu128 \
+    --index-strategy unsafe-best-match
+
+# TESTING: install xformers from source to test 2.7.0 final RC
+RUN --mount=type=cache,target=/root/.cache/uv \
path: docker/Dockerfile
user: login: drisspg
body: If the intention is that the we are just going to wait till xformers releases a 12.8 compat whl though ignore above
created_at: 2025-04-23T17:28:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -163,15 +165,17 @@ FROM base as dev
 # This timeout (in seconds) is necessary when installing some dependencies via uv since it's likely to time out
 # Reference: https://github.com/astral-sh/uv/pull/1694
 ENV UV_HTTP_TIMEOUT=500
+ENV UV_INDEX_STRATEGY="unsafe-best-match"
+
+# Workaround for #17068
+RUN --mount=type=cache,target=/root/.cache/uv \
+    uv pip install --system --no-build-isolation "git+https://github.com/state-spaces/mamba@main"
path: docker/Dockerfile
user: login: mgoin
body: Why install from mamba main?
created_at: 2025-04-24T19:18:19Z
author_association: MEMBER
diff_hunk: @@ -220,7 +225,14 @@ RUN --mount=type=cache,target=/root/.cache/uv \
 # Install vllm wheel first, so that torch etc will be installed.
 RUN --mount=type=bind,from=build,src=/workspace/dist,target=/vllm-workspace/dist \
     --mount=type=cache,target=/root/.cache/uv \
-    uv pip install --system dist/*.whl --verbose
+    uv pip install --system dist/*.whl --verbose \
+    --extra-index-url https://download.pytorch.org/whl/test/cu128 \
+    --index-strategy unsafe-best-match
+
+# TESTING: install xformers from source to test 2.7.0 final RC
+RUN --mount=type=cache,target=/root/.cache/uv \
path: docker/Dockerfile
user: login: huydhn
body: IMO, I think we can just build the package from source for CI, then switch to the official xformers package once it's ready
created_at: 2025-04-25T10:11:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -220,7 +225,14 @@ RUN --mount=type=cache,target=/root/.cache/uv \
 # Install vllm wheel first, so that torch etc will be installed.
 RUN --mount=type=bind,from=build,src=/workspace/dist,target=/vllm-workspace/dist \
     --mount=type=cache,target=/root/.cache/uv \
-    uv pip install --system dist/*.whl --verbose
+    uv pip install --system dist/*.whl --verbose \
+    --extra-index-url https://download.pytorch.org/whl/test/cu128 \
+    --index-strategy unsafe-best-match
+
+# TESTING: install xformers from source to test 2.7.0 final RC
+RUN --mount=type=cache,target=/root/.cache/uv \
path: docker/Dockerfile
user: login: simon-mo
body: How long does this take to compile xformers? If it's too long I don't want to slow down our CI time for this. 
created_at: 2025-04-26T00:29:46Z
author_association: COLLABORATOR
diff_hunk: @@ -240,12 +256,18 @@ RUN --mount=type=bind,from=build,src=/workspace/dist,target=/vllm-workspace/dist
 RUN --mount=type=cache,target=/root/.cache/uv \
 . /etc/environment && \
 if [ "$TARGETPLATFORM" != "linux/arm64" ]; then \
-    uv pip install --system https://github.com/flashinfer-ai/flashinfer/releases/download/v0.2.1.post2/flashinfer_python-0.2.1.post2+cu124torch2.6-cp38-abi3-linux_x86_64.whl ; \
+    # TESTING: install FlashInfer from source to test 2.7.0 final RC
path: docker/Dockerfile
user: login: simon-mo
body: I would imagine FI has new release by now. 
created_at: 2025-04-26T00:30:16Z
author_association: COLLABORATOR
diff_hunk: @@ -8,7 +8,7 @@ image:
   # -- Image tag
   tag: "latest"
   # -- Container launch command
-  command: ["vllm", "serve", "/data/", "--served-model-name", "opt-125m", "--dtype", "bfloat16", "--host", "0.0.0.0", "--port", "8000"]
+  command: ["vllm", "serve", "/data/", "--served-model-name", "opt-125m", "--dtype", "float32", "--block-size", "16", "--host", "0.0.0.0", "--port", "8000"]
path: examples/online_serving/chart-helm/values.yaml
user: login: simon-mo
body: this need to be reverted i assume
created_at: 2025-04-26T00:30:33Z
author_association: COLLABORATOR
diff_hunk: @@ -220,7 +225,14 @@ RUN --mount=type=cache,target=/root/.cache/uv \
 # Install vllm wheel first, so that torch etc will be installed.
 RUN --mount=type=bind,from=build,src=/workspace/dist,target=/vllm-workspace/dist \
     --mount=type=cache,target=/root/.cache/uv \
-    uv pip install --system dist/*.whl --verbose
+    uv pip install --system dist/*.whl --verbose \
+    --extra-index-url https://download.pytorch.org/whl/test/cu128 \
+    --index-strategy unsafe-best-match
+
+# TESTING: install xformers from source to test 2.7.0 final RC
+RUN --mount=type=cache,target=/root/.cache/uv \
path: docker/Dockerfile
user: login: huydhn
body: Let me dig out the number for this once the build finish.  Without caching, it would be significant from what I see locally, but let's see what it takes to build once this is cached.
created_at: 2025-04-26T00:52:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -8,7 +8,7 @@ image:
   # -- Image tag
   tag: "latest"
   # -- Container launch command
-  command: ["vllm", "serve", "/data/", "--served-model-name", "opt-125m", "--dtype", "bfloat16", "--host", "0.0.0.0", "--port", "8000"]
+  command: ["vllm", "serve", "/data/", "--served-model-name", "opt-125m", "--dtype", "float32", "--block-size", "16", "--host", "0.0.0.0", "--port", "8000"]
path: examples/online_serving/chart-helm/values.yaml
user: login: huydhn
body: This is a curious issue that I have seen on CI where cpu build fails on that dtype.  I look around and see a similar issue which suggests switching to `float32` https://github.com/vllm-project/vllm/issues/11327#issuecomment-2774440315.  Let me try to reproduce it to get the full log.
created_at: 2025-04-26T02:00:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -8,7 +8,7 @@ image:
   # -- Image tag
   tag: "latest"
   # -- Container launch command
-  command: ["vllm", "serve", "/data/", "--served-model-name", "opt-125m", "--dtype", "bfloat16", "--host", "0.0.0.0", "--port", "8000"]
+  command: ["vllm", "serve", "/data/", "--served-model-name", "opt-125m", "--dtype", "float32", "--block-size", "16", "--host", "0.0.0.0", "--port", "8000"]
path: examples/online_serving/chart-helm/values.yaml
user: login: huydhn
body: Here is the full server log when serving `vllm serve facebook/opt-125m` on the CPU docker image https://paste.sh/rDYEmrB9#9UuSY3jL6pHBE4MVy_lUDLle.  The change you see here is only to make it work on CI, let me cut an issue https://github.com/vllm-project/vllm/issues/17225
created_at: 2025-04-26T04:09:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -223,7 +235,15 @@ RUN --mount=type=cache,target=/root/.cache/uv \
 # Install vllm wheel first, so that torch etc will be installed.
 RUN --mount=type=bind,from=build,src=/workspace/dist,target=/vllm-workspace/dist \
     --mount=type=cache,target=/root/.cache/uv \
-    uv pip install --system dist/*.whl --verbose
+    uv pip install --system dist/*.whl --verbose \
+    --extra-index-url https://download.pytorch.org/whl/cu128
path: docker/Dockerfile
user: login: mgoin
body: It seems like this should changed based on `CUDA_VERSION`, as this won't work for CUDA_VERSION=12.6
created_at: 2025-04-26T15:38:08Z
author_association: MEMBER
diff_hunk: @@ -6,8 +6,11 @@ numba == 0.61.2; python_version > '3.9'
 
 # Dependencies for NVIDIA GPUs
 ray[cgraph]>=2.43.0, !=2.44.* # Ray Compiled Graph, required for pipeline parallelism in V1.
-torch==2.6.0
-torchaudio==2.6.0
+--extra-index-url https://download.pytorch.org/whl/cu128
path: requirements/cuda.txt
user: login: mgoin
body: ditto on hardcoding cu128 here
created_at: 2025-04-26T15:39:09Z
author_association: MEMBER
diff_hunk: @@ -223,7 +235,15 @@ RUN --mount=type=cache,target=/root/.cache/uv \
 # Install vllm wheel first, so that torch etc will be installed.
 RUN --mount=type=bind,from=build,src=/workspace/dist,target=/vllm-workspace/dist \
     --mount=type=cache,target=/root/.cache/uv \
-    uv pip install --system dist/*.whl --verbose
+    uv pip install --system dist/*.whl --verbose \
+    --extra-index-url https://download.pytorch.org/whl/cu128
path: docker/Dockerfile
user: login: huydhn
body: Yeah, you're right, this needs to change depending on `CUDA_VERSION`, let me add that check to cover 11.8, 12.6, and 12.8 matching those from PyTorch 2.7.0
created_at: 2025-04-26T18:30:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -6,8 +6,11 @@ numba == 0.61.2; python_version > '3.9'
 
 # Dependencies for NVIDIA GPUs
 ray[cgraph]>=2.43.0, !=2.44.* # Ray Compiled Graph, required for pipeline parallelism in V1.
-torch==2.6.0
-torchaudio==2.6.0
+--extra-index-url https://download.pytorch.org/whl/cu128
path: requirements/cuda.txt
user: login: huydhn
body: I'll move the check to the Dockerfile to match the CUDA_VERSION there
created_at: 2025-04-26T18:31:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -77,6 +77,11 @@ ENV TORCH_CUDA_ARCH_LIST=${torch_cuda_arch_list}
 # Override the arch list for flash-attn to reduce the binary size
 ARG vllm_fa_cmake_gpu_arches='80-real;90-real'
 ENV VLLM_FA_CMAKE_GPU_ARCHES=${vllm_fa_cmake_gpu_arches}
+
+# TESTING: install xformers from source to test 2.7.0 final RC
+RUN --mount=type=cache,target=/root/.cache/uv \
+    TORCH_CUDA_ARCH_LIST='7.5 8.0 8.6 8.9 9.0+PTX' \
path: docker/Dockerfile
user: login: cliffwoolley
body: Yes 10.0 as well as 12.0.  +PTX is not enough.  (Same comment in the several places this sequence appears.)  cc @kushanam 
created_at: 2025-04-30T01:00:26Z
author_association: NONE
diff_hunk: @@ -77,6 +77,11 @@ ENV TORCH_CUDA_ARCH_LIST=${torch_cuda_arch_list}
 # Override the arch list for flash-attn to reduce the binary size
 ARG vllm_fa_cmake_gpu_arches='80-real;90-real'
 ENV VLLM_FA_CMAKE_GPU_ARCHES=${vllm_fa_cmake_gpu_arches}
+
+# TESTING: install xformers from source to test 2.7.0 final RC
+RUN --mount=type=cache,target=/root/.cache/uv \
+    TORCH_CUDA_ARCH_LIST='7.5 8.0 8.6 8.9 9.0+PTX' \
path: docker/Dockerfile
user: login: huydhn
body: As the size of this PR is relative big already and its signals are ready, let me add this change to a subsequent PR. It would be clearer that way I guess
created_at: 2025-04-30T01:05:09Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-04-18T16:55:16Z
body: 👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run `fastcheck` CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your `fastcheck` build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping `simon-mo` or `khluu` to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add `ready` label to the PR or enable auto-merge.

🚀
author_association: NONE
user: login: mergify[bot]
created_at: 2025-04-18T16:55:44Z
body: This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @huydhn.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork
author_association: CONTRIBUTOR
user: login: mgoin
created_at: 2025-04-23T17:17:13Z
body: Now that torch 2.7 has released (https://pypi.org/project/torch/2.7.0/) can this be updated?
author_association: MEMBER
user: login: huydhn
created_at: 2025-04-23T17:26:41Z
body: > Now that torch 2.7 has released (https://pypi.org/project/torch/2.7.0/) can this be updated?

Yup, it can be updated now.  Let me start working on that.  On the other hand, I think I will keep the state of this PR as a reference because we plan to do similar validation for the next PyTorch release.  Ideally, the validation needs to be done with PyTorch RC before publishing to pypi.
author_association: CONTRIBUTOR
user: login: huydhn
created_at: 2025-04-27T17:12:35Z
body: @youkaichao @zou3519 I think some of the failed tests are pointing to a real torch.compile compatibility issue with 2.7.0 https://buildkite.com/vllm/ci/builds/18733#01967697-2cf2-469f-91a7-63e66f589d77/205-1262

I can reproduce it with `pytest -v tests/models/test_transformers.py -k test_models[meta-llama/Llama-3.2-1B-Instruct-transformers]`

https://paste.sh/qgshOYtT#6dx7jO1lRsFmoSN43N6aAXeE

Any thoughts?
author_association: CONTRIBUTOR
user: login: zou3519
created_at: 2025-04-28T15:42:35Z
body: vLLM monkeypatches Dynamo to provide a list of file paths that it traced over. Somehow the monkeypatched function provided the filename `"<string>"`. I'll try to run this and see, my best guess right now is that we need to filter out things that are not filepaths
author_association: COLLABORATOR
user: login: zou3519
created_at: 2025-04-28T16:08:30Z
body: After I worked around it (by filtering out `"<string>"`) there are more issues. Currently getting:
```
torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['input_ids'].size()[0],
 L['positions'].size()[0])! For more information, run with TORCH_LOGS="+dynamic".
  - Not all values of RelaxedUnspecConstraint(L['input_ids'].size()[0]) are valid because L['input_ids'].size()
[0] was inferred to be a constant (16384).
  - Not all values of RelaxedUnspecConstraint(L['positions'].size()[0]) are valid because L['positions'].size()
[0] was inferred to be a constant (16384).
```
author_association: COLLABORATOR
user: login: huydhn
created_at: 2025-04-28T17:34:20Z
body: Sync with @zou3519 offline:

* There are at least 2 problems as mentioned in previous comments.  @zou3519 is working on a fix, so the plan is to have that landed, then I can rebase this change
* Maybe the recent transformers bump is related https://github.com/vllm-project/vllm/pull/17116 as the failure seems to come from there
author_association: CONTRIBUTOR
user: login: huydhn
created_at: 2025-04-29T09:41:55Z
body: @zou3519 Thank you for the fix! The failures with `torch.compile` has been resolved.

@simon-mo I think this is ready to land.  The remaining failures on CI are:

* 2 failed speculative decode tests, which I think are existing failures from trunk and are not related to this update.  I attempt to fix them separately at https://github.com/vllm-project/vllm/pull/17371.
* On the other hand, the python only installation test failure is legit, but I think it can only be fixed by updating the nightly binaries.  I have a PR lining up to do that after this land https://github.com/vllm-project/vllm/pull/17224 (need a rebase).  Let me know if that's the regular procedure.

If they are ok, please help land this PR and we can finally have vLLM on 2.7.0 :)
author_association: CONTRIBUTOR
user: login: zou3519
created_at: 2025-04-29T12:03:26Z
body: Btw, did you need https://github.com/vllm-project/vllm/pull/17338 or was that just because I'm on torch > 2.7?
author_association: COLLABORATOR
user: login: mgoin
created_at: 2025-04-29T15:07:50Z
body: New release of xformers with torch 2.7 support has come out https://github.com/facebookresearch/xformers/releases/tag/v0.0.30
author_association: MEMBER
user: login: huydhn
created_at: 2025-04-29T16:49:03Z
body: > Btw, did you need #17338 or was that just because I'm on torch > 2.7?

I don't see any failures due to #17338 on CI, so I guess it's only applicable to PyTorch nightly, not 2.7.0 release.  On the other hand, if you can land your PR before this PR is merged.  I can do a rebase just to be sure.


author_association: CONTRIBUTOR
user: login: zou3519
created_at: 2025-04-29T17:30:44Z
body: > > Btw, did you need #17338 or was that just because I'm on torch > 2.7?
> 
> I don't see any failures due to #17338 on CI, so I guess it's only applicable to PyTorch nightly, not 2.7.0 release. On the other hand, if you can land your PR before this PR is merged. I can do a rebase just to be sure.

No need to wait for my PR if the signals on this PR are good
author_association: COLLABORATOR
user: login: DogeFlow
created_at: 2025-04-30T03:43:50Z
body: When to sync to pip repository
author_association: NONE
user: login: vadimkantorov
created_at: 2025-05-04T19:19:44Z
body: Does vllm publish nightlies to some pip channel? Asking to try out vllm with pytorch 2.7.0
author_association: NONE
user: login: mgoin
created_at: 2025-05-04T19:25:01Z
body: @vadimkantorov 
```
pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
```
author_association: MEMBER
user: login: vadimkantorov
created_at: 2025-05-05T06:36:56Z
body: Couldn't find this nightly instruction in the installation section of README. Might be good to add it there too!
author_association: NONE
user: login: simon-mo
created_at: 2025-05-08T17:54:31Z
body: <!-- linear:replyTo:2824998530 -->

[https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#pre-built-wheels](https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#pre-built-wheels)
author_association: COLLABORATOR
user: login: vadimkantorov
created_at: 2025-05-12T10:59:45Z
body: > pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly

For me, this again tries to fetch 2.6.0 :( despite having 2.7.0 installed:

`vllm-0.8.5.post1-cp38-abi3-manylinux1_x86_64.whl` does not look like nightly wheel :( looks like it's the release wheel. I think pip discovers this wheel from my local cache because it exists and does not attempt to install the nightly :( Bug in pip? I manually went to https://wheels.vllm.ai/nightly/vllm and found this wheel https://wheels.vllm.ai/vllm-0.8.5.dev599%2Bg9fbf2bfbd-cp38-abi3-manylinux1_x86_64.whl - which indeed looks like nightly, but it gives 404 :(

```
$ pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
                                                                                                          
Defaulting to user installation because normal site-packages is not writeable                                              
Looking in indexes: https://pypi.org/simple, https://wheels.vllm.ai/nightly                                                
Collecting vllm                                                                                                              
Downloading vllm-0.8.5.post1-cp38-abi3-manylinux1_x86_64.whl (326.4 MB)                                                        
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 326.4/326.4 MB 10.2 MB/s eta 0:00:00
...
Collecting torch==2.6.0
  Downloading torch-2.6.0-cp310-cp310-manylinux1_x86_64.whl (766.7 MB)
     ━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 221.3/766.7 MB 252.7 MB/s eta 0:00:03                                        
ERROR: Operation cancelled by user

# second try:
$ pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/vllm                                                                                                          
Defaulting to user installation because normal site-packages is not writeable                                              
Looking in indexes: https://pypi.org/simple, https://wheels.vllm.ai/nightly/vllm                                           
Collecting vllm
  Using cached vllm-0.8.5.post1-cp38-abi3-manylinux1_x86_64.whl (326.4 MB)
```



author_association: NONE
user: login: vadimkantorov
created_at: 2025-05-12T11:33:23Z
body: Basically, can't find direct URLs to nightly wheels :( which might be needed to circumvent pip not wanting to install the nightly for some reason

So far managed to find published commit from:
```
pip index versions vllm --pre --extra-index-url https://wheels.vllm.ai/nightly

# and then

pip install -U vllm==0.8.5.dev600+g7ea6cb28b --pre --extra-index-url https://wheels.vllm.ai/nightly
```
author_association: NONE
user: login: zhanglianjie-163
created_at: 2025-06-10T03:02:32Z
body: Hi，why upgrade to pytorch2.7?
author_association: NONE
user: login: jessiewiswjc
created_at: 2025-06-25T03:39:07Z
body: I found that after this commit, the tpot/itl of the Qwen/Qwen2.5-14B-Instruct model on an h20 dropped from 20ms to 10ms. I want to know which part of the code benefited from this. Is it pytorch2.7?
author_association: NONE