Repository: vllm-project/vllm

Issue:
[Bug]: Incorrect Example for the Inference with Prefix 
### Your current environment

```text
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.29.3
Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-97-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.2.140
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100 80GB PCIe
GPU 1: NVIDIA A100 80GB PCIe
GPU 2: NVIDIA A100 80GB PCIe
GPU 3: NVIDIA A100 80GB PCIe

Nvidia driver version: 535.161.07
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      48 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             48
On-line CPU(s) list:                0-47
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 7443 24-Core Processor
CPU family:                         25
Model:                              1
Thread(s) per core:                 1
Core(s) per socket:                 24
Socket(s):                          2
Stepping:                           1
Frequency boost:                    enabled
CPU max MHz:                        4035,6440
CPU min MHz:                        1500,0000
BogoMIPS:                           5700.55
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca
Virtualization:                     AMD-V
L1d cache:                          1,5 MiB (48 instances)
L1i cache:                          1,5 MiB (48 instances)
L2 cache:                           24 MiB (48 instances)
L3 cache:                           256 MiB (8 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0-23
NUMA node1 CPU(s):                  24-47
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] flake8==7.0.0
[pip3] flake8-polyfill==1.0.2
[pip3] mypy-protobuf==3.4.0
[pip3] numpy==1.26.4
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] torch==2.3.0
[pip3] torchinfo==1.8.0
[pip3] torchvision==0.18.0
[pip3] triton==2.3.0
[pip3] vllm_nccl_cu11==2.18.1.0.4.0
[pip3] vllm-nccl-cu12==2.18.1.0.4.0
[conda] Could not collectROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.4.3
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    GPU2    GPU3    NIC0    NIC1    NIC2    NIC3    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NODE    SYS     SYS     PHB     PHB     SYS     SYS     0-23    0               N/A
GPU1    NODE     X      SYS     SYS     NODE    NODE    SYS     SYS     0-23    0               N/A
GPU2    SYS     SYS      X      NODE    SYS     SYS     NODE    NODE    24-47   1               N/A
GPU3    SYS     SYS     NODE     X      SYS     SYS     NODE    NODE    24-47   1               N/A
NIC0    PHB     NODE    SYS     SYS      X      PIX     SYS     SYS
NIC1    PHB     NODE    SYS     SYS     PIX      X      SYS     SYS
NIC2    SYS     SYS     NODE    NODE    SYS     SYS      X      PIX
NIC3    SYS     SYS     NODE    NODE    SYS     SYS     PIX      X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
```


### 🐛 Describe the bug

Currently existing contents of the `vllm/examples/offline_inference_with_prefix.py` have notable issues, introduced during the implementation of `--enable_prefix_caching` in #2762 and its fixes in subsequent PRs. Point of this issue is to display the detected issues and provide a solid ground for upcoming PR to fix them.

---
### 1. Missleading inference speed comparison
**Intended use**: file structure suggests, that example should contain comparion of inference `w/o prefix_caching` vs `w/ prefix_caching`, comparing the model inference speed for those two options.  
**Issue**: for the currently existing implementation, `llm` object is introduced once with `enable_prefix_caching=True`. After that, it is used for generation two times, and those generations are compared with each other.  
As a result, both generations apply `prefix hashing`, and **_resulting inference speed comparison is irrelevant_**. 

```python
# [current state of a file]
from vllm import LLM, SamplingParams

prefix = ...
prompts = [...]

sampling_params = SamplingParams(temperature=0.0)

llm = LLM(model="facebook/opt-125m", enable_prefix_caching=True)
generating_prompts = [prefix + prompt for prompt in prompts]

# [intended generation w/o prefix_caching]
outputs = llm.generate(generating_prompts, sampling_params)

...
# [intended generation w/ prefix_caching]

# The llm.generate call will batch all prompts and send the batch at once
# if resources allow. The prefix will only be cached after the first batch
# is processed, so we need to call generate once to calculate the prefix
# and cache it.
outputs = llm.generate(generating_prompts[0], sampling_params)

# Subsequent batches can leverage the cached prefix
outputs = llm.generate(generating_prompts, sampling_params)

```

**Proposed solution**: create two llm objects, named `regular_llm` and `prefix_cached_llm`, that are `w/o prefix_caching` and `w/ prefix_caching` respectively. Then, run the generation using those two models, obtaining varying results.

---
### 2. Missleading information
**Intended use**: file should cover usage details and explain how to use `enable_prefix_caching` parameter correctly.
**Issue**: currently, file contents states that _"prefix will only be cached after the first batch is processed"_, however my tests disprove that statement, you could test that using the following minimal reproducing code snippet:
```python
# [my code to show that cache works on the fly]
from vllm import LLM, SamplingParams
from time import time

prefix = (
    "You are an expert school principal, skilled in effectively managing "
    "faculty and staff. Draft 10-15 questions for a potential first grade "
    "Head Teacher for my K-12, all-girls', independent school that emphasizes "
    "community, joyful discovery, and life-long learning. The candidate is "
    "coming in for a first-round panel interview for a 8th grade Math "
    "teaching role. They have 5 years of previous teaching experience "
    "as an assistant teacher at a co-ed, public school with experience "
    "in middle school math teaching. Based on these information, fulfill "
    "the following paragraph: ")

prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]

generating_prompts = [prefix + prompt for prompt in prompts]

sampling_params = SamplingParams(temperature=0.0)
regular_llm = LLM(model="facebook/opt-125m", 
                  gpu_memory_utilization=0.4)

prefix_cached_llm = LLM(model="facebook/opt-125m", 
                        gpu_memory_utilization=0.4,
                        enable_prefix_caching=True)

# execution time "without prefix caching"
s = time()
outputs = regular_llm.generate(generating_prompts, sampling_params)
slow_gen_time = time() - s

# execution time when "prefix is not cached yet" 
# according to what is written in example file now
s = time()
outputs = prefix_cached_llm.generate(generating_prompts, sampling_params)
first_gen_time = time() - s

# execution time when "prefix is already cached" 
s = time()
outputs = prefix_cached_llm.generate(generating_prompts, sampling_params)
second_gen_time = time() - s

print(round(first_gen_time / second_gen_time, 2))
print(round(slow_gen_time / second_gen_time, 2))
```

For my setup: 
* speedup for "without prefix caching" vs "prefix is already cached" is `1.3`, 
* speedup for "prefix is already cached" vs "prefix is not cached yet" is `1.01`.   

As there is no significant speedup for the second comparison above, I assume that prefix is cached on the fly during the first answer generation by the model. Therefore, the comment on the "warmup" is missleading and it should be deleted. 
**Proposed solution**: delete missleading comment on the "warmup" generation for prefix caching.


Discussion:

Resolving patch:
diff --git a/examples/offline_inference_with_prefix.py b/examples/offline_inference_with_prefix.py
--- a/examples/offline_inference_with_prefix.py
+++ b/examples/offline_inference_with_prefix.py
@@ -1,5 +1,8 @@
+from time import time
+
 from vllm import LLM, SamplingParams
 
+# Common prefix.
 prefix = (
     "You are an expert school principal, skilled in effectively managing "
     "faculty and staff. Draft 10-15 questions for a potential first grade "
@@ -18,36 +21,60 @@
     "The capital of France is",
     "The future of AI is",
 ]
+
+generating_prompts = [prefix + prompt for prompt in prompts]
+
 # Create a sampling params object.
 sampling_params = SamplingParams(temperature=0.0)
 
 # Create an LLM.
-llm = LLM(model="facebook/opt-125m", enable_prefix_caching=True)
+regular_llm = LLM(model="facebook/opt-125m", gpu_memory_utilization=0.4)
 
-generating_prompts = [prefix + prompt for prompt in prompts]
+prefix_cached_llm = LLM(model="facebook/opt-125m",
+                        enable_prefix_caching=True,
+                        gpu_memory_utilization=0.4)
+print("Results without `enable_prefix_caching`")
 
 # Generate texts from the prompts. The output is a list of RequestOutput objects
 # that contain the prompt, generated text, and other information.
-outputs = llm.generate(generating_prompts, sampling_params)
+start_time_regular = time()
+outputs = regular_llm.generate(generating_prompts, sampling_params)
+duration_regular = time() - start_time_regular
+
+regular_generated_texts = []
 # Print the outputs.
 for output in outputs:
     prompt = output.prompt
     generated_text = output.outputs[0].text
+    regular_generated_texts.append(generated_text)
     print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
 
 print("-" * 80)
 
 # The llm.generate call will batch all prompts and send the batch at once
-# if resources allow. The prefix will only be cached after the first batch
-# is processed, so we need to call generate once to calculate the prefix
-# and cache it.
-outputs = llm.generate(generating_prompts[0], sampling_params)
+# if resources allow.
+start_time_cached = time()
+outputs = prefix_cached_llm.generate(generating_prompts, sampling_params)
+duration_cached = time() - start_time_cached
 
-# Subsequent batches can leverage the cached prefix
-outputs = llm.generate(generating_prompts, sampling_params)
+print("Results with `enable_prefix_caching`")
 
-# Print the outputs. You should see the same outputs as before
+cached_generated_texts = []
+# Print the outputs. You should see the same outputs as before.
 for output in outputs:
     prompt = output.prompt
     generated_text = output.outputs[0].text
+    cached_generated_texts.append(generated_text)
     print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
+
+print("-" * 80)
+
+# Compare the results and display the speedup
+generated_same = all([
+    regular_generated_texts[i] == cached_generated_texts[i]
+    for i in range(len(prompts))
+])
+print(f"Generated answers are the same: {generated_same}")
+
+speedup = round(duration_regular / duration_cached, 2)
+print(f"Speed up of cached generation compared to the regular is: {speedup}")


Test patch: