The model's max seq len (128000) is larger than the maximum number of tokens that can be stored in KV cache (78336). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.

### Reminder

- [x] I have read the above rules and searched the existing issues.

### System Info


- `llamafactory` version: 0.9.2.dev0
- Platform: Linux-5.15.0-127-generic-x86_64-with-glibc2.31
- Python version: 3.10.16
- PyTorch version: 2.5.1+cu124 (GPU)
- Transformers version: 4.49.0
- Datasets version: 3.2.0
- Accelerate version: 1.2.1
- PEFT version: 0.12.0
- TRL version: 0.9.6
- GPU type: NVIDIA GeForce RTX 4090
- GPU number: 4
- GPU memory: 23.64GB
- vLLM version: 0.7.3

### Reproduction

```bash
[WARNING|logging.py:329] 2025-03-10 21:43:12,326 >> It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
(VllmWorkerProcess pid=921093) It looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.
[WARNING|tokenization_utils_base.py:3945] 2025-03-10 21:44:27,701 >> Token indices sequence length is longer than the specified maximum sequence length for this model (131072 > 2048). Running this sequence through the model will result in indexing errors
(VllmWorkerProcess pid=921093) Token indices sequence length is longer than the specified maximum sequence length for this model (131072 > 2048). Running this sequence through the model will result in indexing errors
WARNING 03-10 21:44:36 profiling.py:192] The context length (128000) of the model is too short to hold the multi-modal embeddings in the worst case (131072 tokens in total, out of which {'image': 65536, 'video': 65536} are reserved for multi-modal embeddings). This may cause certain multi-modal inputs to fail during inference, even when the input text is short. To avoid this, you should increase `max_model_len`, reduce `max_num_seqs`, and/or reduce `mm_counts`.
(VllmWorkerProcess pid=921093) WARNING 03-10 21:44:37 profiling.py:192] The context length (128000) of the model is too short to hold the multi-modal embeddings in the worst case (131072 tokens in total, out of which {'image': 65536, 'video': 65536} are reserved for multi-modal embeddings). This may cause certain multi-modal inputs to fail during inference, even when the input text is short. To avoid this, you should increase `max_model_len`, reduce `max_num_seqs`, and/or reduce `mm_counts`.
(VllmWorkerProcess pid=921093) INFO 03-10 21:45:04 worker.py:267] Memory profiling takes 138.53 seconds
(VllmWorkerProcess pid=921093) INFO 03-10 21:45:04 worker.py:267] the current vLLM instance can use total_gpu_memory (23.64GiB) x gpu_memory_utilization (0.90) = 21.28GiB
(VllmWorkerProcess pid=921093) INFO 03-10 21:45:04 worker.py:267] model weights take 7.87GiB; non_torch_memory takes 0.25GiB; PyTorch activation peak memory takes 11.06GiB; the rest of the memory reserved for KV Cache is 2.09GiB.
INFO 03-10 21:45:04 worker.py:267] Memory profiling takes 138.69 seconds
INFO 03-10 21:45:04 worker.py:267] the current vLLM instance can use total_gpu_memory (23.64GiB) x gpu_memory_utilization (0.90) = 21.28GiB
INFO 03-10 21:45:04 worker.py:267] model weights take 7.87GiB; non_torch_memory takes 0.25GiB; PyTorch activation peak memory takes 11.06GiB; the rest of the memory reserved for KV Cache is 2.09GiB.
INFO 03-10 21:45:04 executor_base.py:111] # cuda blocks: 4896, # CPU blocks: 9362
INFO 03-10 21:45:04 executor_base.py:116] Maximum concurrency for 128000 tokens per request: 0.61x
(VllmWorkerProcess pid=921093) ERROR 03-10 21:45:04 multiproc_worker_utils.py:242] Exception in worker VllmWorkerProcess while processing method initialize_cache.
(VllmWorkerProcess pid=921093) ERROR 03-10 21:45:04 multiproc_worker_utils.py:242] Traceback (most recent call last):
(VllmWorkerProcess pid=921093) ERROR 03-10 21:45:04 multiproc_worker_utils.py:242]   File "/data1/user/md/anaconda3/envs/factory/lib/python3.10/site-packages/vllm/executor/multiproc_worker_utils.py", line 236, in _run_worker_process
(VllmWorkerProcess pid=921093) ERROR 03-10 21:45:04 multiproc_worker_utils.py:242]     output = run_method(worker, method, args, kwargs)
(VllmWorkerProcess pid=921093) ERROR 03-10 21:45:04 multiproc_worker_utils.py:242]   File "/data1/user/md/anaconda3/envs/factory/lib/python3.10/site-packages/vllm/utils.py", line 2196, in run_method
(VllmWorkerProcess pid=921093) ERROR 03-10 21:45:04 multiproc_worker_utils.py:242]     return func(*args, **kwargs)
(VllmWorkerProcess pid=921093) ERROR 03-10 21:45:04 multiproc_worker_utils.py:242]   File "/data1/user/md/anaconda3/envs/factory/lib/python3.10/site-packages/vllm/worker/worker.py", line 291, in initialize_cache
(VllmWorkerProcess pid=921093) ERROR 03-10 21:45:04 multiproc_worker_utils.py:242]     raise_if_cache_size_invalid(num_gpu_blocks,
(VllmWorkerProcess pid=921093) ERROR 03-10 21:45:04 multiproc_worker_utils.py:242]   File "/data1/user/md/anaconda3/envs/factory/lib/python3.10/site-packages/vllm/worker/worker.py", line 544, in raise_if_cache_size_invalid
(VllmWorkerProcess pid=921093) ERROR 03-10 21:45:04 multiproc_worker_utils.py:242]     raise ValueError(
(VllmWorkerProcess pid=921093) ERROR 03-10 21:45:04 multiproc_worker_utils.py:242] ValueError: The model's max seq len (128000) is larger than the maximum number of tokens that can be stored in KV cache (78336). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
[rank0]: Traceback (most recent call last):
[rank0]:   File "/data1/user/md/workspace/NTIRE/yb_LLaMA-Factory/scripts/vllm_infer.py", line 147, in <module>
[rank0]:     fire.Fire(vllm_infer)
[rank0]:   File "/data1/user/md/anaconda3/envs/factory/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
[rank0]:     component_trace = _Fire(component, args, parsed_flag_args, context, name)
[rank0]:   File "/data1/user/md/anaconda3/envs/factory/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
[rank0]:     component, remaining_args = _CallAndUpdateTrace(
[rank0]:   File "/data1/user/md/anaconda3/envs/factory/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
[rank0]:     component = fn(*varargs, **kwargs)
[rank0]:   File "/data1/user/md/workspace/NTIRE/yb_LLaMA-Factory/scripts/vllm_infer.py", line 123, in vllm_infer
[rank0]:     results = LLM(**engine_args).generate(inputs, sampling_params, lora_request=lora_request)
[rank0]:   File "/data1/user/md/anaconda3/envs/factory/lib/python3.10/site-packages/vllm/utils.py", line 1022, in inner
[rank0]:     return fn(*args, **kwargs)
[rank0]:   File "/data1/user/md/anaconda3/envs/factory/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 242, in __init__
[rank0]:     self.llm_engine = self.engine_class.from_engine_args(
[rank0]:   File "/data1/user/md/anaconda3/envs/factory/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 489, in from_engine_args
[rank0]:     engine = cls(
[rank0]:   File "/data1/user/md/anaconda3/envs/factory/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 276, in __init__
[rank0]:     self._initialize_kv_caches()
[rank0]:   File "/data1/user/md/anaconda3/envs/factory/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 434, in _initialize_kv_caches
[rank0]:     self.model_executor.initialize_cache(num_gpu_blocks, num_cpu_blocks)
[rank0]:   File "/data1/user/md/anaconda3/envs/factory/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 122, in initialize_cache
[rank0]:     self.collective_rpc("initialize_cache",
[rank0]:   File "/data1/user/md/anaconda3/envs/factory/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 316, in collective_rpc
[rank0]:     return self._run_workers(method, *args, **(kwargs or {}))
[rank0]:   File "/data1/user/md/anaconda3/envs/factory/lib/python3.10/site-packages/vllm/executor/mp_distributed_executor.py", line 185, in _run_workers
[rank0]:     driver_worker_output = run_method(self.driver_worker, sent_method,
[rank0]:   File "/data1/user/md/anaconda3/envs/factory/lib/python3.10/site-packages/vllm/utils.py", line 2196, in run_method
[rank0]:     return func(*args, **kwargs)
[rank0]:   File "/data1/user/md/anaconda3/envs/factory/lib/python3.10/site-packages/vllm/worker/worker.py", line 291, in initialize_cache
[rank0]:     raise_if_cache_size_invalid(num_gpu_blocks,
[rank0]:   File "/data1/user/md/anaconda3/envs/factory/lib/python3.10/site-packages/vllm/worker/worker.py", line 544, in raise_if_cache_size_invalid
[rank0]:     raise ValueError(
[rank0]: ValueError: The model's max seq len (128000) is larger than the maximum number of tokens that can be stored in KV cache (78336). Try increasing `gpu_memory_utilization` or decreasing `max_model_len` when initializing the engine.
INFO 03-10 21:45:05 multiproc_worker_utils.py:128] Killing local vLLM worker processes
Fatal Python error: _enter_buffered_busy: could not acquire lock for <_io.BufferedWriter name='<stdout>'> at interpreter shutdown, possibly due to daemon threads
Python runtime state: finalizing (tstate=0x000000003201f040)

Current thread 0x00007fd51f6cc280 (most recent call first):
  <no Python frame>

Extension modules: charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, yaml._yaml, markupsafe._speedups, PIL._imaging, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, _cffi_backend, pyarrow._parquet, pyarrow._fs, pyarrow._azurefs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, xxhash._xxhash, pyarrow._json, pyarrow._acero, pyarrow._csv, pyarrow._substrait, pyarrow._dataset, pyarrow._dataset_orc, pyarrow._parquet_encryption, pyarrow._dataset_parquet_encryption, pyarrow._dataset_parquet, sklearn.__check_build._check_build, psutil._psutil_linux, psutil._psutil_posix, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy._lib._uarray._uarray, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._dierckx, scipy.interpolate._ppoly, scipy.interpolate._interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.interpolate._bspl, scipy.special.cython_special, scipy.stats._stats, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._biasedurn, scipy.stats._stats_pythran, scipy.stats._levy_stable.levyst, scipy.stats._ansari_swilk_statistics, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.ndimage._nd_image, scipy.ndimage._rank_filter_1d, _ni_label, scipy.ndimage._ni_label, sklearn.utils._isfinite, sklearn.utils.sparsefuncs_fast, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.utils._vector_sentinel, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, PIL._imagingft, av._core, av.logging, av.bytesource, av.buffer, av.audio.format, av.error, av.dictionary, av.container.pyio, av.utils, av.option, av.descriptor, av.format, av.stream, av.container.streams, av.sidedata.motionvectors, av.sidedata.sidedata, av.opaque, av.packet, av.container.input, av.container.output, av.container.core, av.codec.context, av.video.format, av.video.reformatter, av.plane, av.video.plane, av.video.frame, av.video.stream, av.codec.hwaccel, av.codec.codec, av.frame, av.audio.layout, av.audio.plane, av.audio.frame, av.audio.stream, av.filter.pad, av.filter.link, av.filter.context, av.filter.graph, av.filter.filter, av.audio.resampler, av.filter.loudnorm, av.audio.codeccontext, av.audio.fifo, av.bitstream, av.video.codeccontext, msgspec._core, zmq.backend.cython._zmq, msgpack._cmsgpack, google._upb._message, setproctitle, uvloop.loop, ray._raylet, regex._regex, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, PIL._imagingmath, vllm.cumem_allocator, sentencepiece._sentencepiece, cuda_utils, __triton_launcher (total: 266)
/data1/user/md/anaconda3/envs/factory/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/data1/user/md/anaconda3/envs/factory/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
Aborted (core dumped)
```

### Others

My input command line： `CUDA_VISIBLE_DEVICES=2,3 python scripts/vllm_infer.py --model_name_or_path outputs/qwen2_5_vl_lora_sft_md --dataset evalmuse_test/evalmuse_test_0 --template qwen2_vl`

I tried to add `--vllm_maxlen78336` but still failed

Labels: solved
State: closed