Repository: vllm-project/vllm

Issue:
[Bug]: benchmark trtllm failed
### Your current environment

```text
trtllm: 0.8.0
triton server version: 2.43.0
triton_backend version: a4c7ac7e4036a2c7e41ee663697859cc8575367d
tritonserver docker: nvcr.io/nvidia/tritonserver:24.02-trtllm-python-py3
```


### 🐛 Describe the bug

benchmark trtllm using the trt_llm backend with the following command:
```bash
python3 benchmark_serving.py \
               --endpoint /v2/models/ensemble/generate_stream \
               --port 19000 \
               --backend tensorrt-llm \
               --model llama2-7b \
               --tokenizer $HF_MODEL_DIR \
               --dataset-name sharegpt \
               --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json \
               --request-rate 128 \
               --num-prompts 1000
```
I get the error:
```text
prompt_len=29, error='Traceback (most recent call last):\n  File "/tensorrtllm_backend/dataset-benchmark/backend_request_func.py", line 154, in async_request_trt_llm\n    output.generated_text = json.loads(data)["text_output"]\n  File "/usr/lib/python3.10/json/__init__.py", line 339, in loads\n    raise TypeError(f\'the JSON object must be str, bytes or bytearray, \'\nTypeError: the JSON object must be str, bytes or bytearray, not dict\n'),
```
which I think's it's due to the duplicated `json.load` of `data` returned by trtllm
https://github.com/vllm-project/vllm/blob/11d652bd4f6f81d09638399885099b78a4e3b9c8/benchmarks/backend_request_func.py#L137
https://github.com/vllm-project/vllm/blob/11d652bd4f6f81d09638399885099b78a4e3b9c8/benchmarks/backend_request_func.py#L152

And in the streaming mode, response is returned token by token, the final output_text only collect the last chunck of data in the current implementation:
https://github.com/vllm-project/vllm/blob/11d652bd4f6f81d09638399885099b78a4e3b9c8/benchmarks/backend_request_func.py#L152
so the output length is wrong when computing decode latency here:
https://github.com/vllm-project/vllm/blob/11d652bd4f6f81d09638399885099b78a4e3b9c8/benchmarks/benchmark_serving.py#L205

[Installation]: import llm meet error
### Your current environment

```text
Traceback (most recent call last):
  File "inference.py", line 355, in <module>
    data_all_with_response = get_pred_func(data=data_all, task_prompt=task_prompt,\
  File "inference.py", line 24, in get_pred_vllm
    from vllm import LLM, SamplingParams
  File "/usr/local/lib/python3.8/dist-packages/vllm/__init__.py", line 3, in <module>
    from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs
  File "/usr/local/lib/python3.8/dist-packages/vllm/engine/arg_utils.py", line 6, in <module>
    from vllm.config import (CacheConfig, ModelConfig, ParallelConfig,
  File "/usr/local/lib/python3.8/dist-packages/vllm/config.py", line 9, in <module>
    from vllm.utils import get_cpu_memory, is_hip
  File "/usr/local/lib/python3.8/dist-packages/vllm/utils.py", line 8, in <module>
    from vllm._C import cuda_utils
ImportError: /usr/local/lib/python3.8/dist-packages/vllm/_C.cpython-38-x86_64-linux-gnu.so: undefined symbol: _ZN2at4_ops15to_dtype_layout4callERKNS_6TensorEN3c108optionalINS5_10ScalarTypeEEENS6_INS5_6LayoutEEENS6_INS5_6DeviceEEENS6_IbEEbbNS6_INS5_12MemoryFormatEEE```


### How you are installing vllm

```sh
export VLLM_VERSION=0.2.4
export PYTHON_VERSION=38
pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux1_x86_64.whl --extra-index-url https://download.pytorch.org/whl/cu118
```



Discussion:

Resolving patch:
diff --git a/benchmarks/backend_request_func.py b/benchmarks/backend_request_func.py
--- a/benchmarks/backend_request_func.py
+++ b/benchmarks/backend_request_func.py
@@ -135,6 +135,7 @@ async def async_request_trt_llm(
                                               "data:")
 
                         data = json.loads(chunk)
+                        output.generated_text += data["text_output"]
                         timestamp = time.perf_counter()
                         # First token
                         if ttft == 0.0:
@@ -149,7 +150,6 @@ async def async_request_trt_llm(
                         most_recent_timestamp = timestamp
 
                     output.latency = most_recent_timestamp - st
-                    output.generated_text = json.loads(data)["text_output"]
                     output.success = True
 
                 else:


Test patch: