Repository: wenet-e2e/wenet

Issue:
streaming triton error
**Describe the bug**
Decoder onnx model output tensor is empty which cause dl_pack error. @Slyne 

![image](https://user-images.githubusercontent.com/11469548/178891936-c9ec777c-8d67-4114-ac53-f131dfdb2ad7.png)

**To Reproduce**
Just run streaming triton server as runtime/GPU README, no error occured.

client test:
`python3 client.py --audio_file=test.wav --url=localhost:8001 --model_name=streaming_wenet --streaming`

**Expected behavior**
Correct output

**Screenshots**
![image](https://user-images.githubusercontent.com/11469548/178891995-3f627347-deaf-49bd-8e78-523ac1d66572.png)

![image](https://user-images.githubusercontent.com/11469548/178891599-84fc96bc-6368-41d8-9c84-af12888a980f.png)

**Desktop (please complete the following information):**
 - OS: Ubuntu20.04
 - CUDA: 11.3
 - tritonserver: 20.03


Discussion:
Are all the test audios causing the same error ?

Could you help check the server side and show the log/screen output on server side?
Not sure if decoder has been set up correctly.

Please also provide a test audio file if that is possible.
> Are all the test audios causing the same error ?
> 
> Could you help check the server side and show the log/screen output on server side? Not sure if decoder has been set up correctly.
> 
> Please also provide a test audio file if that is possible.

1. yes
2. emmm, I also think it's quite possible, so may you have a look of my server log, thanks.
https://drive.google.com/file/d/1_arhDIkYAM3ZjND0ClhWuFRjWjMZFh9n/view?usp=sharing
> > Are all the test audios causing the same error ?
> > Could you help check the server side and show the log/screen output on server side? Not sure if decoder has been set up correctly.
> > Please also provide a test audio file if that is possible.
> 
> 1. yes
> 2. emmm, I also think it's quite possible, so may you have a look of my server log, thanks.
>    https://drive.google.com/file/d/1_arhDIkYAM3ZjND0ClhWuFRjWjMZFh9n/view?usp=sharing

Could you try to trace back the error as early as possible? Maybe you could print the inputs of self.batch_rescoring call, see if  the score_hyps is empty or not.
> > > Are all the test audios causing the same error ?
> > > Could you help check the server side and show the log/screen output on server side? Not sure if decoder has been set up correctly.
> > > Please also provide a test audio file if that is possible.
> > 
> > 
> > 
> > 1. yes
> > 2. emmm, I also think it's quite possible, so may you have a look of my server log, thanks.
> >    https://drive.google.com/file/d/1_arhDIkYAM3ZjND0ClhWuFRjWjMZFh9n/view?usp=sharing
> 
> Could you try to trace back the error as early as possible? Maybe you could print the inputs of self.batch_rescoring call, see if the score_hyps is empty or not.

thanks for you reply, score_hyps has data:

![image](https://user-images.githubusercontent.com/11469548/179199990-c024773e-7506-4cf7-9531-449982942725.png)

print code
```py
print("=> rescore_hyps:\n", rescore_hyps)
print("=> rescore_encoder_hist:\n", rescore_encoder_hist)
print("=> rescore_encoder_lens:\n", rescore_encoder_lens)
print("=> max_length:\n", max_length)
best_index = self.batch_rescoring(rescore_hyps, rescore_encoder_hist,
                                  rescore_encoder_lens, max_length)
```
> > > > Are all the test audios causing the same error ?
> > > > Could you help check the server side and show the log/screen output on server side? Not sure if decoder has been set up correctly.
> > > > Please also provide a test audio file if that is possible.
> > > 
> > > 
> > > 
> > > 1. yes
> > > 2. emmm, I also think it's quite possible, so may you have a look of my server log, thanks.
> > >    https://drive.google.com/file/d/1_arhDIkYAM3ZjND0ClhWuFRjWjMZFh9n/view?usp=sharing
> > 
> > 
> > Could you try to trace back the error as early as possible? Maybe you could print the inputs of self.batch_rescoring call, see if the score_hyps is empty or not.
> 
> thanks for you reply, score_hyps has data:
> 
> ![image](https://user-images.githubusercontent.com/11469548/179199990-c024773e-7506-4cf7-9531-449982942725.png)
> 
> print code
> 
> ```python
> print("=> rescore_hyps:\n", rescore_hyps)
> print("=> rescore_encoder_hist:\n", rescore_encoder_hist)
> print("=> rescore_encoder_lens:\n", rescore_encoder_lens)
> print("=> max_length:\n", max_length)
> best_index = self.batch_rescoring(rescore_hyps, rescore_encoder_hist,
>                                   rescore_encoder_lens, max_length)
> ```

Thanks. Could you please give me a minimal essentials to reproduce the issue? Maybe your dockerfile , model_repo configs (if you modified any of them), include one test audio would be better.
> > > > > Are all the test audios causing the same error ?
> > > > > Could you help check the server side and show the log/screen output on server side? Not sure if decoder has been set up correctly.
> > > > > Please also provide a test audio file if that is possible.
> > > > 
> > > > 
> > > > 
> > > > 1. yes
> > > > 2. emmm, I also think it's quite possible, so may you have a look of my server log, thanks.
> > > >    https://drive.google.com/file/d/1_arhDIkYAM3ZjND0ClhWuFRjWjMZFh9n/view?usp=sharing
> > > 
> > > 
> > > Could you try to trace back the error as early as possible? Maybe you could print the inputs of self.batch_rescoring call, see if the score_hyps is empty or not.
> > 
> > 
> > thanks for you reply, score_hyps has data:
> > ![image](https://user-images.githubusercontent.com/11469548/179199990-c024773e-7506-4cf7-9531-449982942725.png)
> > print code
> > ```python
> > print("=> rescore_hyps:\n", rescore_hyps)
> > print("=> rescore_encoder_hist:\n", rescore_encoder_hist)
> > print("=> rescore_encoder_lens:\n", rescore_encoder_lens)
> > print("=> max_length:\n", max_length)
> > best_index = self.batch_rescoring(rescore_hyps, rescore_encoder_hist,
> >                                   rescore_encoder_lens, max_length)
> > ```
> 
> Thanks. Could you please give me a minimal essentials to reproduce the issue? Maybe your dockerfile , model_repo configs (if you modified any of them), include one test audio would be better.

OK, I will prepare these data today, thank you.
@yuekaizhang I reproduce this problem by these steps:

1. build and run the docker file(both server and client in one container)
```sh
FROM nvcr.io/nvidia/tritonserver:22.03-py3
LABEL maintainer="NVIDIA"
LABEL repository="tritonserver"

# fix public key error
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
RUN dpkg -i cuda-keyring_1.0-1_all.deb

RUN apt-get update 
RUN apt-get install -fy cmake make swig
RUN pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113
RUN pip3 install -v kaldifeat
RUN pip3 install pyyaml onnx onnxruntime-gpu typeguard 

# for client
RUN apt-get install -y libsndfile1
RUN pip3 install soundfile grpcio-tools tritonclient

WORKDIR /workspace
RUN git clone https://github.com/Slyne/ctc_decoder.git && cd ctc_decoder/swig && bash setup.sh
COPY ./scripts scripts
```
run docker container:

```sh
docker run --name wenet-triton --gpus '"device=3"' -it --shm-size=1g --ulimit memlock=-1 -v <wenet-dir>:/ws/wenet -v <20210618_u2pp_conformer_exp dir>:/ws/model wenet-tritonserver:22.03-py3 /bin/bash
```

the 20210618_u2pp_conformer_exp was downloaded from wenet github repo.


2. into the container

``` sh
export model_dir=/ws/model
export PYTHONPATH=$PYTHONPATH:/ws/wenet
```
export onnx model

```sh
cd /ws/wenet/
python3 wenet/bin/export_onnx_gpu.py --config=$model_dir/train.yaml --checkpoint=$model_dir/final.pt --cmvn_file=$model_dir/global_cmvn --ctc_weight=0.3 --reverse_weight=0.3 --output_onnx_dir=$model_dir/onnx_gpu --streaming
```

convert config files

```sh
cd /ws/wenet/runtime/GPU
python3 scripts/convert.py --config=$model_dir/train.yaml --vocab=$model_dir/words.txt --model_repo=/ws/wenet/runtime/GPU/model_repo_stateful/ --onnx_model_dir=$model_dir/onnx_gpu
```

start server

```sh
cd /ws/wenet/runtime/GPU
tritonserver --model-repository=/ws/wenet/runtime/GPU/model_repo_stateful/ --pinned-memory-pool-byte-size=1024000000 --cuda-memory-pool-byte-size=0:1024000000
```

run client test in another terminal
```sh
cd /ws/wenet/runtime/GPU
python3 client.py --audio_file=chinese_test.wav --url=localhost:8001 --model_name=streaming_wenet --streaming
```

audio file:
https://drive.google.com/file/d/1xsodix-CW9dDHZn0pFEnpVMSfyngFbh1/view?usp=sharing

If you encounter any problems during these operation, please contact me, thank you.
> @yuekaizhang I reproduce this problem by these steps:
> 
> 1. build and run the docker file(both server and client in one container)
> 
> ```shell
> FROM nvcr.io/nvidia/tritonserver:22.03-py3
> LABEL maintainer="NVIDIA"
> LABEL repository="tritonserver"
> 
> # fix public key error
> RUN wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-keyring_1.0-1_all.deb
> RUN dpkg -i cuda-keyring_1.0-1_all.deb
> 
> RUN apt-get update 
> RUN apt-get install -fy cmake make swig
> RUN pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113
> RUN pip3 install -v kaldifeat
> RUN pip3 install pyyaml onnx onnxruntime-gpu typeguard 
> 
> # for client
> RUN apt-get install -y libsndfile1
> RUN pip3 install soundfile grpcio-tools tritonclient
> 
> WORKDIR /workspace
> RUN git clone https://github.com/Slyne/ctc_decoder.git && cd ctc_decoder/swig && bash setup.sh
> COPY ./scripts scripts
> ```
> 
> run docker container:
> 
> ```shell
> docker run --name wenet-triton --gpus '"device=3"' -it --shm-size=1g --ulimit memlock=-1 -v <wenet-dir>:/ws/wenet -v <20210618_u2pp_conformer_exp dir>:/ws/model wenet-tritonserver:22.03-py3 /bin/bash
> ```
> 
> the 20210618_u2pp_conformer_exp was downloaded from wenet github repo.
> 
> 2. into the container
> 
> ```shell
> export model_dir=/ws/model
> export PYTHONPATH=$PYTHONPATH:/ws/wenet
> ```
> 
> export onnx model
> 
> ```shell
> cd /ws/wenet/
> python3 wenet/bin/export_onnx_gpu.py --config=$model_dir/train.yaml --checkpoint=$model_dir/final.pt --cmvn_file=$model_dir/global_cmvn --ctc_weight=0.3 --reverse_weight=0.3 --output_onnx_dir=$model_dir/onnx_gpu --streaming
> ```
> 
> convert config files
> 
> ```shell
> cd /ws/wenet/runtime/GPU
> python3 scripts/convert.py --config=$model_dir/train.yaml --vocab=$model_dir/words.txt --model_repo=/ws/wenet/runtime/GPU/model_repo_stateful/ --onnx_model_dir=$model_dir/onnx_gpu
> ```
> 
> start server
> 
> ```shell
> cd /ws/wenet/runtime/GPU
> tritonserver --model-repository=/ws/wenet/runtime/GPU/model_repo_stateful/ --pinned-memory-pool-byte-size=1024000000 --cuda-memory-pool-byte-size=0:1024000000
> ```
> 
> run client test in another terminal
> 
> ```shell
> cd /ws/wenet/runtime/GPU
> python3 client.py --audio_file=chinese_test.wav --url=localhost:8001 --model_name=streaming_wenet --streaming
> ```
> 
> audio file: https://drive.google.com/file/d/1xsodix-CW9dDHZn0pFEnpVMSfyngFbh1/view?usp=sharing
> 
> If you encounter any problems during these operation, please contact me, thank you.

Reproduced it, thx. I will debug it tomorrow. I recommend you use --fp16 when export to onnx , which works for me.

Resolving patch:
diff --git a/runtime/GPU/model_repo_stateful/wenet/1/model.py b/runtime/GPU/model_repo_stateful/wenet/1/model.py
--- a/runtime/GPU/model_repo_stateful/wenet/1/model.py
+++ b/runtime/GPU/model_repo_stateful/wenet/1/model.py
@@ -57,7 +57,7 @@ def initialize(self, args):
             self.device = 'cpu'
 
         # get parameter configurations
-        self.model = WenetModel(model_config["parameters"], self.device)
+        self.model = WenetModel(self.model_config, self.device)
 
         # Get OUTPUT0 configuration
         output0_config = pb_utils.get_output_config_by_name(
diff --git a/runtime/GPU/model_repo_stateful/wenet/1/wenet_onnx_model.py b/runtime/GPU/model_repo_stateful/wenet/1/wenet_onnx_model.py
--- a/runtime/GPU/model_repo_stateful/wenet/1/wenet_onnx_model.py
+++ b/runtime/GPU/model_repo_stateful/wenet/1/wenet_onnx_model.py
@@ -23,7 +23,7 @@
 
 class WenetModel(object):
     def __init__(self, model_config, device):
-        params = self.parse_model_parameters(model_config)
+        params = self.parse_model_parameters(model_config['parameters'])
 
         self.device = device
         print("Using device", device)
@@ -55,7 +55,16 @@ def __init__(self, model_config, device):
         print("Using rescoring:", bool(self.rescoring))
         print("Successfully load all parameters!")
 
-        self.dtype = torch.float16
+        log_probs_config = pb_utils.get_input_config_by_name(
+            model_config, "log_probs")
+        # Convert Triton types to numpy types
+        log_probs_dtype = pb_utils.triton_string_to_numpy(
+            log_probs_config['data_type'])
+
+        if log_probs_dtype == np.float32:
+            self.dtype = torch.float32
+        else:
+            self.dtype = torch.float16
 
     def generate_init_cache(self):
         encoder_out = None


Test patch: