Repository: espnet/espnet

Issue:
Sometimes IndexError is happened in BatchBeamSearch of ESPnet2 
**Describe the bug**
Sometimes IndexError is happend in BatchBeamSearch of ESPnet2 during decoding.

**Basic environments:**
- OS information: Linux 3.10.0-1062.9.1.el7.x86_64 #1 SMP Mon Dec 2 08:31:54 EST 2019 x86_64
- python version: `3.7.3 (default, Mar 27 2019, 22:11:17)  [GCC 7.3.0]`
- espnet version: `espnet 0.8.0`
- pytorch version: `pytorch 1.5.1`
- Git hash: `bdf0fde7252ef371879c708ca1d9e465eec6d442`
  - Commit date: `Sun Sep 13 22:53:12 2020 +0900`

**Environments from `torch.utils.collect_env`:**

```
PyTorch version: 1.5.1
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Red Hat Enterprise Linux Server release 7.7 (Maipo)
GCC version: (GCC) 8.4.0
CMake version: version 3.17.3

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.2.89
GPU models and configuration:
GPU 0: Tesla V100-PCIE-32GB
GPU 1: Tesla V100-PCIE-32GB

Nvidia driver version: 440.64.00
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.18.5
[pip3] pytorch-ranger==0.1.1
[pip3] pytorch-wpe==0.0.0
[pip3] torch==1.5.1
[pip3] torch-complex==0.1.0
[pip3] torch-optimizer==0.0.1a14
[pip3] warprnnt-pytorch==0.1
[conda] blas                      1.0                         mkl
[conda] mkl                       2020.1                      217
[conda] mkl-service               2.3.0            py37he904b0f_0
[conda] mkl_fft                   1.1.0            py37h23d657b_0
[conda] mkl_random                1.1.1            py37h0573a6f_0
[conda] pytorch                   1.5.1           py3.7_cuda10.1.243_cudnn7.6.3_0    pytorch
[conda] pytorch-ranger            0.1.1                    pypi_0    pypi
[conda] pytorch-wpe               0.0.0                    pypi_0    pypi
[conda] torch-complex             0.1.0                    pypi_0    pypi
[conda] torch-optimizer           0.0.1a14                 pypi_0    pypi
[conda] warprnnt-pytorch          0.1                      pypi_0    pypi
```

**Task information:**
 - Task: ASR
 - Recipe: CSJ
 - ESPnet2

**To Reproduce**
Steps to reproduce the behavior:

1. Run decoding state `./run.sh --stage 11`

**Error logs**

```
# python3 -m espnet2.bin.asr_inference --ngpu 0 --data_path_and_name_and_type dump/raw/eval1/wav.scp,speech,sound --key_file exp/asr_train_asr_transformer_raw_char_sp/decode_asr_lm_lm_train_lm_char_valid.loss.ave_asr_model_valid.acc.best/eval1/logdir/keys.10.scp --asr_train_config exp/asr_train_asr_transformer_raw_char_sp/config.yaml --asr_model_file exp/asr_train_asr_transformer_raw_char_sp/valid.acc.best.pth --output_dir exp/asr_train_asr_transformer_raw_char_sp/decode_asr_lm_lm_train_lm_char_valid.loss.ave_asr_model_valid.acc.best/eval1/logdir/output.10 --config conf/decode_asr.yaml --lm_train_config exp/lm_train_lm_char/config.yaml --lm_file exp/lm_train_lm_char/valid.loss.ave.pth 
# Started at Wed Sep 16 11:49:54 JST 2020
#
/home/z43614a/work/espnet/tools/venv/bin/python3 /home/z43614a/work/espnet/espnet2/bin/asr_inference.py --ngpu 0 --data_path_and_name_and_type dump/raw/eval1/wav.scp,speech,sound --key_file exp/asr_train_asr_transformer_raw_char_sp/decode_asr_lm_lm_train_lm_char_valid.loss.ave_asr_model_valid.acc.best/eval1/logdir/keys.10.scp --asr_train_config exp/asr_train_asr_transformer_raw_char_sp/config.yaml --asr_model_file exp/asr_train_asr_transformer_raw_char_sp/valid.acc.best.pth --output_dir exp/asr_train_asr_transformer_raw_char_sp/decode_asr_lm_lm_train_lm_char_valid.loss.ave_asr_model_valid.acc.best/eval1/logdir/output.10 --config conf/decode_asr.yaml --lm_train_config exp/lm_train_lm_char/config.yaml --lm_file exp/lm_train_lm_char/valid.loss.ave.pth
2020-09-16 11:51:07,297 (asr:281) INFO: Vocabulary size: 3262
2020-09-16 11:51:11,029 (lm:198) INFO: Vocabulary size: 3262
2020-09-16 11:51:11,260 (asr_inference:119) INFO: BatchBeamSearch implementation is selected.
2020-09-16 11:51:11,265 (asr_inference:129) INFO: Beam_search: BatchBeamSearch(
  (nn_dict): ModuleDict(
    (decoder): TransformerDecoder(
      (embed): Sequential(
        (0): Embedding(3262, 512)
        (1): PositionalEncoding(
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
      (after_norm): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
      (output_layer): Linear(in_features=512, out_features=3262, bias=True)
      (decoders): MultiSequential(
        (0): DecoderLayer(
          (self_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (src_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (feed_forward): PositionwiseFeedForward(
            (w_1): Linear(in_features=512, out_features=2048, bias=True)
            (w_2): Linear(in_features=2048, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
            (activation): ReLU()
          )
          (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
        (1): DecoderLayer(
          (self_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (src_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (feed_forward): PositionwiseFeedForward(
            (w_1): Linear(in_features=512, out_features=2048, bias=True)
            (w_2): Linear(in_features=2048, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
            (activation): ReLU()
          )
          (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
        (2): DecoderLayer(
          (self_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (src_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (feed_forward): PositionwiseFeedForward(
            (w_1): Linear(in_features=512, out_features=2048, bias=True)
            (w_2): Linear(in_features=2048, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
            (activation): ReLU()
          )
          (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
        (3): DecoderLayer(
          (self_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (src_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (feed_forward): PositionwiseFeedForward(
            (w_1): Linear(in_features=512, out_features=2048, bias=True)
            (w_2): Linear(in_features=2048, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
            (activation): ReLU()
          )
          (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
        (4): DecoderLayer(
          (self_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (src_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (feed_forward): PositionwiseFeedForward(
            (w_1): Linear(in_features=512, out_features=2048, bias=True)
            (w_2): Linear(in_features=2048, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
            (activation): ReLU()
          )
          (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
        (5): DecoderLayer(
          (self_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (src_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (feed_forward): PositionwiseFeedForward(
            (w_1): Linear(in_features=512, out_features=2048, bias=True)
            (w_2): Linear(in_features=2048, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
            (activation): ReLU()
          )
          (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
    )
    (lm): SequentialRNNLM(
      (drop): Dropout(p=0.0, inplace=False)
      (encoder): Embedding(3262, 650, padding_idx=0)
      (rnn): LSTM(650, 650, num_layers=2, batch_first=True)
      (decoder): Linear(in_features=650, out_features=3262, bias=True)
    )
  )
)
2020-09-16 11:51:11,265 (asr_inference:130) INFO: Decoding device=cpu, dtype=float32
2020-09-16 11:51:11,322 (asr_inference:148) INFO: Text tokenizer: CharTokenizer(space_symbol="<space>"non_linguistic_symbols="set()")
2020-09-16 11:51:13,432 (beam_search:358) INFO: decoder input length: 156
2020-09-16 11:51:13,433 (beam_search:359) INFO: max output length: 156
2020-09-16 11:51:13,433 (beam_search:360) INFO: min output length: 0
/opt/conda/conda-bld/pytorch_1591914880026/work/torch/csrc/utils/python_arg_parser.cpp:756: UserWarning: This overload of nonzero is deprecated:
	nonzero(Tensor input, *, Tensor out)
Consider using one of the following signatures instead:
	nonzero(Tensor input, *, bool as_tuple)
2020-09-16 11:51:36,910 (beam_search:372) INFO: end detected at 65
2020-09-16 11:51:36,911 (beam_search:397) INFO:  -5.10 * 0.7 =  -3.57 for decoder
2020-09-16 11:51:36,911 (beam_search:397) INFO:  -6.90 * 0.3 =  -2.07 for ctc
2020-09-16 11:51:36,911 (beam_search:397) INFO: -154.84 * 0.3 = -46.45 for lm
2020-09-16 11:51:36,911 (beam_search:399) INFO: total log probability: -52.09
2020-09-16 11:51:36,911 (beam_search:400) INFO: normalized log probability: -0.85
2020-09-16 11:51:36,911 (beam_search:401) INFO: total number of ended hypotheses: 79
2020-09-16 11:51:36,911 (beam_search:406) INFO: best hypo: 同じ制御で構文解析ができるとでこの場合重要なのはこのアルゴリズムがオートマトンの具体形に依存しないという点にございます

2020-09-16 11:51:38,032 (beam_search:358) INFO: decoder input length: 222
2020-09-16 11:51:38,033 (beam_search:359) INFO: max output length: 222
2020-09-16 11:51:38,033 (beam_search:360) INFO: min output length: 0
2020-09-16 11:52:25,634 (beam_search:372) INFO: end detected at 101
2020-09-16 11:52:25,635 (beam_search:397) INFO:  -9.83 * 0.7 =  -6.88 for decoder
2020-09-16 11:52:25,636 (beam_search:397) INFO: -11.97 * 0.3 =  -3.59 for ctc
2020-09-16 11:52:25,636 (beam_search:397) INFO: -201.95 * 0.3 = -60.59 for lm
2020-09-16 11:52:25,636 (beam_search:399) INFO: total log probability: -71.06
2020-09-16 11:52:25,636 (beam_search:400) INFO: normalized log probability: -0.74
2020-09-16 11:52:25,636 (beam_search:401) INFO: total number of ended hypotheses: 81
2020-09-16 11:52:25,636 (beam_search:406) INFO: best hypo: でこのような文法形式においてじゃどういう風な言語を表現できるのかということをま簡単にまとめときますと例えばえーっとまーコンテキストフィーランゲージですねまヘッドの概念はえーと必要なんですが

2020-09-16 11:52:26,598 (beam_search:358) INFO: decoder input length: 209
2020-09-16 11:52:26,598 (beam_search:359) INFO: max output length: 209
2020-09-16 11:52:26,598 (beam_search:360) INFO: min output length: 0
2020-09-16 11:53:02,819 (beam_search:372) INFO: end detected at 80
2020-09-16 11:53:02,820 (beam_search:397) INFO: -12.95 * 0.7 =  -9.07 for decoder
2020-09-16 11:53:02,820 (beam_search:397) INFO: -18.95 * 0.3 =  -5.68 for ctc
2020-09-16 11:53:02,820 (beam_search:397) INFO: -214.31 * 0.3 = -64.29 for lm
2020-09-16 11:53:02,820 (beam_search:399) INFO: total log probability: -79.05
2020-09-16 11:53:02,820 (beam_search:400) INFO: normalized log probability: -1.07
2020-09-16 11:53:02,820 (beam_search:401) INFO: total number of ended hypotheses: 69
2020-09-16 11:53:02,821 (beam_search:406) INFO: best hypo: そういったものは実は先のスパインに沿って様子ツリーを分解しますとえーっと生起言語を作ることが分かりますので有限状態をトマトンに表現できるあるいは

2020-09-16 11:53:03,856 (beam_search:358) INFO: decoder input length: 230
2020-09-16 11:53:03,856 (beam_search:359) INFO: max output length: 230
2020-09-16 11:53:03,856 (beam_search:360) INFO: min output length: 0
2020-09-16 11:53:58,540 (beam_search:372) INFO: end detected at 113
2020-09-16 11:53:58,541 (beam_search:397) INFO: -18.32 * 0.7 = -12.83 for decoder
2020-09-16 11:53:58,541 (beam_search:397) INFO: -39.31 * 0.3 = -11.79 for ctc
2020-09-16 11:53:58,541 (beam_search:397) INFO: -293.19 * 0.3 = -87.96 for lm
2020-09-16 11:53:58,541 (beam_search:399) INFO: total log probability: -112.58
2020-09-16 11:53:58,542 (beam_search:400) INFO: normalized log probability: -1.05
2020-09-16 11:53:58,542 (beam_search:401) INFO: total number of ended hypotheses: 74
2020-09-16 11:53:58,542 (beam_search:406) INFO: best hypo: えっとまクリアージョニングランゲージですねこれはスパイによって分解しますと様子ツリーの推定変数というのがあーの文脈自由言語なすということが知られていますのでこう下のオートマトンで表現ができるという特徴があります

2020-09-16 11:53:58,932 (beam_search:358) INFO: decoder input length: 81
2020-09-16 11:53:58,932 (beam_search:359) INFO: max output length: 81
2020-09-16 11:53:58,932 (beam_search:360) INFO: min output length: 0
2020-09-16 11:54:10,401 (beam_search:372) INFO: end detected at 53
2020-09-16 11:54:10,402 (beam_search:397) INFO:  -9.26 * 0.7 =  -6.48 for decoder
2020-09-16 11:54:10,402 (beam_search:397) INFO: -17.02 * 0.3 =  -5.10 for ctc
2020-09-16 11:54:10,402 (beam_search:397) INFO: -132.29 * 0.3 = -39.69 for lm
2020-09-16 11:54:10,402 (beam_search:399) INFO: total log probability: -51.28
2020-09-16 11:54:10,402 (beam_search:400) INFO: normalized log probability: -1.09
2020-09-16 11:54:10,402 (beam_search:401) INFO: total number of ended hypotheses: 78
2020-09-16 11:54:10,403 (beam_search:406) INFO: best hypo: オートマトンのクラスがよく下がっているのはツリーを対比してる方はクラスイック下がる訳ですね

2020-09-16 11:54:11,404 (beam_search:358) INFO: decoder input length: 235
2020-09-16 11:54:11,404 (beam_search:359) INFO: max output length: 235
2020-09-16 11:54:11,404 (beam_search:360) INFO: min output length: 0
2020-09-16 11:54:59,835 (beam_search:372) INFO: end detected at 108
2020-09-16 11:54:59,836 (beam_search:397) INFO: -15.15 * 0.7 = -10.60 for decoder
2020-09-16 11:54:59,836 (beam_search:397) INFO: -21.38 * 0.3 =  -6.42 for ctc
2020-09-16 11:54:59,836 (beam_search:397) INFO: -248.58 * 0.3 = -74.57 for lm
2020-09-16 11:54:59,836 (beam_search:399) INFO: total log probability: -91.59
2020-09-16 11:54:59,836 (beam_search:400) INFO: normalized log probability: -0.91
2020-09-16 11:54:59,836 (beam_search:401) INFO: total number of ended hypotheses: 85
2020-09-16 11:54:59,837 (beam_search:406) INFO: best hypo: でえっとーまそういう風な特徴があってえーとん文脈依存言語までもえーと表現できるという特徴がありますでここで後これプラッカーに意味があるかとかよく分かんないんですけどオートマトンが単語とに可変ですので

2020-09-16 11:54:59,973 (beam_search:358) INFO: decoder input length: 19
2020-09-16 11:54:59,973 (beam_search:359) INFO: max output length: 19
2020-09-16 11:54:59,973 (beam_search:360) INFO: min output length: 0
2020-09-16 11:55:01,277 (beam_search:372) INFO: end detected at 15
2020-09-16 11:55:01,278 (beam_search:397) INFO:  -1.13 * 0.7 =  -0.79 for decoder
2020-09-16 11:55:01,278 (beam_search:397) INFO:  -0.26 * 0.3 =  -0.08 for ctc
2020-09-16 11:55:01,278 (beam_search:397) INFO: -21.19 * 0.3 =  -6.36 for lm
2020-09-16 11:55:01,278 (beam_search:399) INFO: total log probability: -7.23
2020-09-16 11:55:01,278 (beam_search:400) INFO: normalized log probability: -0.72
2020-09-16 11:55:01,278 (beam_search:401) INFO: total number of ended hypotheses: 82
2020-09-16 11:55:01,278 (beam_search:406) INFO: best hypo: あるとこにあのー

2020-09-16 11:55:01,904 (beam_search:358) INFO: decoder input length: 152
2020-09-16 11:55:01,904 (beam_search:359) INFO: max output length: 152
2020-09-16 11:55:01,904 (beam_search:360) INFO: min output length: 0
2020-09-16 11:55:28,522 (beam_search:372) INFO: end detected at 87
2020-09-16 11:55:28,523 (beam_search:397) INFO: -10.70 * 0.7 =  -7.49 for decoder
2020-09-16 11:55:28,523 (beam_search:397) INFO: -12.28 * 0.3 =  -3.68 for ctc
2020-09-16 11:55:28,523 (beam_search:397) INFO: -222.42 * 0.3 = -66.73 for lm
2020-09-16 11:55:28,523 (beam_search:399) INFO: total log probability: -77.90
2020-09-16 11:55:28,523 (beam_search:400) INFO: normalized log probability: -0.96
2020-09-16 11:55:28,523 (beam_search:401) INFO: total number of ended hypotheses: 80
2020-09-16 11:55:28,523 (beam_search:406) INFO: best hypo: 今までコンテストリーランゲーで書いていたものにちょっとだけコンテストセンティブを導入するといったようなあの単語ごとにクラスを可変にするといったような操作も可能

2020-09-16 11:55:28,622 (beam_search:358) INFO: decoder input length: 8
2020-09-16 11:55:28,622 (beam_search:359) INFO: max output length: 8
2020-09-16 11:55:28,622 (beam_search:360) INFO: min output length: 0
2020-09-16 11:55:29,079 (batch_beam_search:321) INFO: adding <eos> in the last position in the loop
Traceback (most recent call last):
  File "/home/kan-bayashi/work/espnet/tools/venv/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kan-bayashi/work/espnet/tools/venv/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/kan-bayashi/work/espnet/espnet2/bin/asr_inference.py", line 447, in <module>
    main()
  File "/home/kan-bayashi/work/espnet/espnet2/bin/asr_inference.py", line 443, in main
    inference(**kwargs)
  File "/home/kan-bayashi/work/espnet/espnet2/bin/asr_inference.py", line 312, in inference
    results = speech2text(**batch)
  File "/home/kan-bayashi/work/espnet/tools/venv/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/home/kan-bayashi/work/espnet/espnet2/bin/asr_inference.py", line 194, in __call__
    x=enc[0], maxlenratio=self.maxlenratio, minlenratio=self.minlenratio
  File "/home/kan-bayashi/work/espnet/tools/venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/kan-bayashi/work/espnet/espnet/nets/beam_search.py", line 369, in forward
    running_hyps = self.post_process(i, maxlen, maxlenratio, best, ended_hyps)
  File "/home/kan-bayashi/work/espnet/espnet/nets/batch_beam_search.py", line 341, in post_process
    running_hyps.yseq[torch.arange(n_batch), running_hyps.length - 1]
IndexError: index 19 is out of bounds for dimension 1 with size 10
# Accounting: time=335 threads=1
# Ended (code 1) at Wed Sep 16 11:55:29 JST 2020, elapsed time 335 seconds
```

Sometimes IndexError is happened in BatchBeamSearch of ESPnet2 
**Describe the bug**
Sometimes IndexError is happend in BatchBeamSearch of ESPnet2 during decoding.

**Basic environments:**
- OS information: Linux 3.10.0-1062.9.1.el7.x86_64 #1 SMP Mon Dec 2 08:31:54 EST 2019 x86_64
- python version: `3.7.3 (default, Mar 27 2019, 22:11:17)  [GCC 7.3.0]`
- espnet version: `espnet 0.8.0`
- pytorch version: `pytorch 1.5.1`
- Git hash: `bdf0fde7252ef371879c708ca1d9e465eec6d442`
  - Commit date: `Sun Sep 13 22:53:12 2020 +0900`

**Environments from `torch.utils.collect_env`:**

```
PyTorch version: 1.5.1
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Red Hat Enterprise Linux Server release 7.7 (Maipo)
GCC version: (GCC) 8.4.0
CMake version: version 3.17.3

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.2.89
GPU models and configuration:
GPU 0: Tesla V100-PCIE-32GB
GPU 1: Tesla V100-PCIE-32GB

Nvidia driver version: 440.64.00
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.18.5
[pip3] pytorch-ranger==0.1.1
[pip3] pytorch-wpe==0.0.0
[pip3] torch==1.5.1
[pip3] torch-complex==0.1.0
[pip3] torch-optimizer==0.0.1a14
[pip3] warprnnt-pytorch==0.1
[conda] blas                      1.0                         mkl
[conda] mkl                       2020.1                      217
[conda] mkl-service               2.3.0            py37he904b0f_0
[conda] mkl_fft                   1.1.0            py37h23d657b_0
[conda] mkl_random                1.1.1            py37h0573a6f_0
[conda] pytorch                   1.5.1           py3.7_cuda10.1.243_cudnn7.6.3_0    pytorch
[conda] pytorch-ranger            0.1.1                    pypi_0    pypi
[conda] pytorch-wpe               0.0.0                    pypi_0    pypi
[conda] torch-complex             0.1.0                    pypi_0    pypi
[conda] torch-optimizer           0.0.1a14                 pypi_0    pypi
[conda] warprnnt-pytorch          0.1                      pypi_0    pypi
```

**Task information:**
 - Task: ASR
 - Recipe: CSJ
 - ESPnet2

**To Reproduce**
Steps to reproduce the behavior:

1. Run decoding state `./run.sh --stage 11`

**Error logs**

```
# python3 -m espnet2.bin.asr_inference --ngpu 0 --data_path_and_name_and_type dump/raw/eval1/wav.scp,speech,sound --key_file exp/asr_train_asr_transformer_raw_char_sp/decode_asr_lm_lm_train_lm_char_valid.loss.ave_asr_model_valid.acc.best/eval1/logdir/keys.10.scp --asr_train_config exp/asr_train_asr_transformer_raw_char_sp/config.yaml --asr_model_file exp/asr_train_asr_transformer_raw_char_sp/valid.acc.best.pth --output_dir exp/asr_train_asr_transformer_raw_char_sp/decode_asr_lm_lm_train_lm_char_valid.loss.ave_asr_model_valid.acc.best/eval1/logdir/output.10 --config conf/decode_asr.yaml --lm_train_config exp/lm_train_lm_char/config.yaml --lm_file exp/lm_train_lm_char/valid.loss.ave.pth 
# Started at Wed Sep 16 11:49:54 JST 2020
#
/home/z43614a/work/espnet/tools/venv/bin/python3 /home/z43614a/work/espnet/espnet2/bin/asr_inference.py --ngpu 0 --data_path_and_name_and_type dump/raw/eval1/wav.scp,speech,sound --key_file exp/asr_train_asr_transformer_raw_char_sp/decode_asr_lm_lm_train_lm_char_valid.loss.ave_asr_model_valid.acc.best/eval1/logdir/keys.10.scp --asr_train_config exp/asr_train_asr_transformer_raw_char_sp/config.yaml --asr_model_file exp/asr_train_asr_transformer_raw_char_sp/valid.acc.best.pth --output_dir exp/asr_train_asr_transformer_raw_char_sp/decode_asr_lm_lm_train_lm_char_valid.loss.ave_asr_model_valid.acc.best/eval1/logdir/output.10 --config conf/decode_asr.yaml --lm_train_config exp/lm_train_lm_char/config.yaml --lm_file exp/lm_train_lm_char/valid.loss.ave.pth
2020-09-16 11:51:07,297 (asr:281) INFO: Vocabulary size: 3262
2020-09-16 11:51:11,029 (lm:198) INFO: Vocabulary size: 3262
2020-09-16 11:51:11,260 (asr_inference:119) INFO: BatchBeamSearch implementation is selected.
2020-09-16 11:51:11,265 (asr_inference:129) INFO: Beam_search: BatchBeamSearch(
  (nn_dict): ModuleDict(
    (decoder): TransformerDecoder(
      (embed): Sequential(
        (0): Embedding(3262, 512)
        (1): PositionalEncoding(
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
      (after_norm): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
      (output_layer): Linear(in_features=512, out_features=3262, bias=True)
      (decoders): MultiSequential(
        (0): DecoderLayer(
          (self_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (src_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (feed_forward): PositionwiseFeedForward(
            (w_1): Linear(in_features=512, out_features=2048, bias=True)
            (w_2): Linear(in_features=2048, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
            (activation): ReLU()
          )
          (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
        (1): DecoderLayer(
          (self_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (src_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (feed_forward): PositionwiseFeedForward(
            (w_1): Linear(in_features=512, out_features=2048, bias=True)
            (w_2): Linear(in_features=2048, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
            (activation): ReLU()
          )
          (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
        (2): DecoderLayer(
          (self_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (src_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (feed_forward): PositionwiseFeedForward(
            (w_1): Linear(in_features=512, out_features=2048, bias=True)
            (w_2): Linear(in_features=2048, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
            (activation): ReLU()
          )
          (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
        (3): DecoderLayer(
          (self_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (src_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (feed_forward): PositionwiseFeedForward(
            (w_1): Linear(in_features=512, out_features=2048, bias=True)
            (w_2): Linear(in_features=2048, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
            (activation): ReLU()
          )
          (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
        (4): DecoderLayer(
          (self_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (src_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (feed_forward): PositionwiseFeedForward(
            (w_1): Linear(in_features=512, out_features=2048, bias=True)
            (w_2): Linear(in_features=2048, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
            (activation): ReLU()
          )
          (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
        (5): DecoderLayer(
          (self_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (src_attn): MultiHeadedAttention(
            (linear_q): Linear(in_features=512, out_features=512, bias=True)
            (linear_k): Linear(in_features=512, out_features=512, bias=True)
            (linear_v): Linear(in_features=512, out_features=512, bias=True)
            (linear_out): Linear(in_features=512, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (feed_forward): PositionwiseFeedForward(
            (w_1): Linear(in_features=512, out_features=2048, bias=True)
            (w_2): Linear(in_features=2048, out_features=512, bias=True)
            (dropout): Dropout(p=0.1, inplace=False)
            (activation): ReLU()
          )
          (norm1): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm2): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (norm3): LayerNorm((512,), eps=1e-12, elementwise_affine=True)
          (dropout): Dropout(p=0.1, inplace=False)
        )
      )
    )
    (lm): SequentialRNNLM(
      (drop): Dropout(p=0.0, inplace=False)
      (encoder): Embedding(3262, 650, padding_idx=0)
      (rnn): LSTM(650, 650, num_layers=2, batch_first=True)
      (decoder): Linear(in_features=650, out_features=3262, bias=True)
    )
  )
)
2020-09-16 11:51:11,265 (asr_inference:130) INFO: Decoding device=cpu, dtype=float32
2020-09-16 11:51:11,322 (asr_inference:148) INFO: Text tokenizer: CharTokenizer(space_symbol="<space>"non_linguistic_symbols="set()")
2020-09-16 11:51:13,432 (beam_search:358) INFO: decoder input length: 156
2020-09-16 11:51:13,433 (beam_search:359) INFO: max output length: 156
2020-09-16 11:51:13,433 (beam_search:360) INFO: min output length: 0
/opt/conda/conda-bld/pytorch_1591914880026/work/torch/csrc/utils/python_arg_parser.cpp:756: UserWarning: This overload of nonzero is deprecated:
	nonzero(Tensor input, *, Tensor out)
Consider using one of the following signatures instead:
	nonzero(Tensor input, *, bool as_tuple)
2020-09-16 11:51:36,910 (beam_search:372) INFO: end detected at 65
2020-09-16 11:51:36,911 (beam_search:397) INFO:  -5.10 * 0.7 =  -3.57 for decoder
2020-09-16 11:51:36,911 (beam_search:397) INFO:  -6.90 * 0.3 =  -2.07 for ctc
2020-09-16 11:51:36,911 (beam_search:397) INFO: -154.84 * 0.3 = -46.45 for lm
2020-09-16 11:51:36,911 (beam_search:399) INFO: total log probability: -52.09
2020-09-16 11:51:36,911 (beam_search:400) INFO: normalized log probability: -0.85
2020-09-16 11:51:36,911 (beam_search:401) INFO: total number of ended hypotheses: 79
2020-09-16 11:51:36,911 (beam_search:406) INFO: best hypo: 同じ制御で構文解析ができるとでこの場合重要なのはこのアルゴリズムがオートマトンの具体形に依存しないという点にございます

2020-09-16 11:51:38,032 (beam_search:358) INFO: decoder input length: 222
2020-09-16 11:51:38,033 (beam_search:359) INFO: max output length: 222
2020-09-16 11:51:38,033 (beam_search:360) INFO: min output length: 0
2020-09-16 11:52:25,634 (beam_search:372) INFO: end detected at 101
2020-09-16 11:52:25,635 (beam_search:397) INFO:  -9.83 * 0.7 =  -6.88 for decoder
2020-09-16 11:52:25,636 (beam_search:397) INFO: -11.97 * 0.3 =  -3.59 for ctc
2020-09-16 11:52:25,636 (beam_search:397) INFO: -201.95 * 0.3 = -60.59 for lm
2020-09-16 11:52:25,636 (beam_search:399) INFO: total log probability: -71.06
2020-09-16 11:52:25,636 (beam_search:400) INFO: normalized log probability: -0.74
2020-09-16 11:52:25,636 (beam_search:401) INFO: total number of ended hypotheses: 81
2020-09-16 11:52:25,636 (beam_search:406) INFO: best hypo: でこのような文法形式においてじゃどういう風な言語を表現できるのかということをま簡単にまとめときますと例えばえーっとまーコンテキストフィーランゲージですねまヘッドの概念はえーと必要なんですが

2020-09-16 11:52:26,598 (beam_search:358) INFO: decoder input length: 209
2020-09-16 11:52:26,598 (beam_search:359) INFO: max output length: 209
2020-09-16 11:52:26,598 (beam_search:360) INFO: min output length: 0
2020-09-16 11:53:02,819 (beam_search:372) INFO: end detected at 80
2020-09-16 11:53:02,820 (beam_search:397) INFO: -12.95 * 0.7 =  -9.07 for decoder
2020-09-16 11:53:02,820 (beam_search:397) INFO: -18.95 * 0.3 =  -5.68 for ctc
2020-09-16 11:53:02,820 (beam_search:397) INFO: -214.31 * 0.3 = -64.29 for lm
2020-09-16 11:53:02,820 (beam_search:399) INFO: total log probability: -79.05
2020-09-16 11:53:02,820 (beam_search:400) INFO: normalized log probability: -1.07
2020-09-16 11:53:02,820 (beam_search:401) INFO: total number of ended hypotheses: 69
2020-09-16 11:53:02,821 (beam_search:406) INFO: best hypo: そういったものは実は先のスパインに沿って様子ツリーを分解しますとえーっと生起言語を作ることが分かりますので有限状態をトマトンに表現できるあるいは

2020-09-16 11:53:03,856 (beam_search:358) INFO: decoder input length: 230
2020-09-16 11:53:03,856 (beam_search:359) INFO: max output length: 230
2020-09-16 11:53:03,856 (beam_search:360) INFO: min output length: 0
2020-09-16 11:53:58,540 (beam_search:372) INFO: end detected at 113
2020-09-16 11:53:58,541 (beam_search:397) INFO: -18.32 * 0.7 = -12.83 for decoder
2020-09-16 11:53:58,541 (beam_search:397) INFO: -39.31 * 0.3 = -11.79 for ctc
2020-09-16 11:53:58,541 (beam_search:397) INFO: -293.19 * 0.3 = -87.96 for lm
2020-09-16 11:53:58,541 (beam_search:399) INFO: total log probability: -112.58
2020-09-16 11:53:58,542 (beam_search:400) INFO: normalized log probability: -1.05
2020-09-16 11:53:58,542 (beam_search:401) INFO: total number of ended hypotheses: 74
2020-09-16 11:53:58,542 (beam_search:406) INFO: best hypo: えっとまクリアージョニングランゲージですねこれはスパイによって分解しますと様子ツリーの推定変数というのがあーの文脈自由言語なすということが知られていますのでこう下のオートマトンで表現ができるという特徴があります

2020-09-16 11:53:58,932 (beam_search:358) INFO: decoder input length: 81
2020-09-16 11:53:58,932 (beam_search:359) INFO: max output length: 81
2020-09-16 11:53:58,932 (beam_search:360) INFO: min output length: 0
2020-09-16 11:54:10,401 (beam_search:372) INFO: end detected at 53
2020-09-16 11:54:10,402 (beam_search:397) INFO:  -9.26 * 0.7 =  -6.48 for decoder
2020-09-16 11:54:10,402 (beam_search:397) INFO: -17.02 * 0.3 =  -5.10 for ctc
2020-09-16 11:54:10,402 (beam_search:397) INFO: -132.29 * 0.3 = -39.69 for lm
2020-09-16 11:54:10,402 (beam_search:399) INFO: total log probability: -51.28
2020-09-16 11:54:10,402 (beam_search:400) INFO: normalized log probability: -1.09
2020-09-16 11:54:10,402 (beam_search:401) INFO: total number of ended hypotheses: 78
2020-09-16 11:54:10,403 (beam_search:406) INFO: best hypo: オートマトンのクラスがよく下がっているのはツリーを対比してる方はクラスイック下がる訳ですね

2020-09-16 11:54:11,404 (beam_search:358) INFO: decoder input length: 235
2020-09-16 11:54:11,404 (beam_search:359) INFO: max output length: 235
2020-09-16 11:54:11,404 (beam_search:360) INFO: min output length: 0
2020-09-16 11:54:59,835 (beam_search:372) INFO: end detected at 108
2020-09-16 11:54:59,836 (beam_search:397) INFO: -15.15 * 0.7 = -10.60 for decoder
2020-09-16 11:54:59,836 (beam_search:397) INFO: -21.38 * 0.3 =  -6.42 for ctc
2020-09-16 11:54:59,836 (beam_search:397) INFO: -248.58 * 0.3 = -74.57 for lm
2020-09-16 11:54:59,836 (beam_search:399) INFO: total log probability: -91.59
2020-09-16 11:54:59,836 (beam_search:400) INFO: normalized log probability: -0.91
2020-09-16 11:54:59,836 (beam_search:401) INFO: total number of ended hypotheses: 85
2020-09-16 11:54:59,837 (beam_search:406) INFO: best hypo: でえっとーまそういう風な特徴があってえーとん文脈依存言語までもえーと表現できるという特徴がありますでここで後これプラッカーに意味があるかとかよく分かんないんですけどオートマトンが単語とに可変ですので

2020-09-16 11:54:59,973 (beam_search:358) INFO: decoder input length: 19
2020-09-16 11:54:59,973 (beam_search:359) INFO: max output length: 19
2020-09-16 11:54:59,973 (beam_search:360) INFO: min output length: 0
2020-09-16 11:55:01,277 (beam_search:372) INFO: end detected at 15
2020-09-16 11:55:01,278 (beam_search:397) INFO:  -1.13 * 0.7 =  -0.79 for decoder
2020-09-16 11:55:01,278 (beam_search:397) INFO:  -0.26 * 0.3 =  -0.08 for ctc
2020-09-16 11:55:01,278 (beam_search:397) INFO: -21.19 * 0.3 =  -6.36 for lm
2020-09-16 11:55:01,278 (beam_search:399) INFO: total log probability: -7.23
2020-09-16 11:55:01,278 (beam_search:400) INFO: normalized log probability: -0.72
2020-09-16 11:55:01,278 (beam_search:401) INFO: total number of ended hypotheses: 82
2020-09-16 11:55:01,278 (beam_search:406) INFO: best hypo: あるとこにあのー

2020-09-16 11:55:01,904 (beam_search:358) INFO: decoder input length: 152
2020-09-16 11:55:01,904 (beam_search:359) INFO: max output length: 152
2020-09-16 11:55:01,904 (beam_search:360) INFO: min output length: 0
2020-09-16 11:55:28,522 (beam_search:372) INFO: end detected at 87
2020-09-16 11:55:28,523 (beam_search:397) INFO: -10.70 * 0.7 =  -7.49 for decoder
2020-09-16 11:55:28,523 (beam_search:397) INFO: -12.28 * 0.3 =  -3.68 for ctc
2020-09-16 11:55:28,523 (beam_search:397) INFO: -222.42 * 0.3 = -66.73 for lm
2020-09-16 11:55:28,523 (beam_search:399) INFO: total log probability: -77.90
2020-09-16 11:55:28,523 (beam_search:400) INFO: normalized log probability: -0.96
2020-09-16 11:55:28,523 (beam_search:401) INFO: total number of ended hypotheses: 80
2020-09-16 11:55:28,523 (beam_search:406) INFO: best hypo: 今までコンテストリーランゲーで書いていたものにちょっとだけコンテストセンティブを導入するといったようなあの単語ごとにクラスを可変にするといったような操作も可能

2020-09-16 11:55:28,622 (beam_search:358) INFO: decoder input length: 8
2020-09-16 11:55:28,622 (beam_search:359) INFO: max output length: 8
2020-09-16 11:55:28,622 (beam_search:360) INFO: min output length: 0
2020-09-16 11:55:29,079 (batch_beam_search:321) INFO: adding <eos> in the last position in the loop
Traceback (most recent call last):
  File "/home/kan-bayashi/work/espnet/tools/venv/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/kan-bayashi/work/espnet/tools/venv/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/kan-bayashi/work/espnet/espnet2/bin/asr_inference.py", line 447, in <module>
    main()
  File "/home/kan-bayashi/work/espnet/espnet2/bin/asr_inference.py", line 443, in main
    inference(**kwargs)
  File "/home/kan-bayashi/work/espnet/espnet2/bin/asr_inference.py", line 312, in inference
    results = speech2text(**batch)
  File "/home/kan-bayashi/work/espnet/tools/venv/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/home/kan-bayashi/work/espnet/espnet2/bin/asr_inference.py", line 194, in __call__
    x=enc[0], maxlenratio=self.maxlenratio, minlenratio=self.minlenratio
  File "/home/kan-bayashi/work/espnet/tools/venv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/kan-bayashi/work/espnet/espnet/nets/beam_search.py", line 369, in forward
    running_hyps = self.post_process(i, maxlen, maxlenratio, best, ended_hyps)
  File "/home/kan-bayashi/work/espnet/espnet/nets/batch_beam_search.py", line 341, in post_process
    running_hyps.yseq[torch.arange(n_batch), running_hyps.length - 1]
IndexError: index 19 is out of bounds for dimension 1 with size 10
# Accounting: time=335 threads=1
# Ended (code 1) at Wed Sep 16 11:55:29 JST 2020, elapsed time 335 seconds
```



Discussion:
https://github.com/espnet/espnet/blob/67ca53d61932e4ff3325f1ea7215eead03f300bb/espnet/nets/batch_beam_search.py#L338-L343
I found the comment.
~~@ShigekiKarita Is it OK to change `torch.arange(n_batch)` -> `torch.arange(len(runnning_hyps))`?~~
I misunderstood the problem. Let me debug more.
The problem is that `yseq.shape` is `(20, 10)` but `length` is `[20,20,20,...]`.
https://github.com/espnet/espnet/blob/67ca53d61932e4ff3325f1ea7215eead03f300bb/espnet/nets/batch_beam_search.py#L336
Maybe here should be `running_hyps.length[:] = yseq_eos.shape[1]`?
https://github.com/espnet/espnet/blob/67ca53d61932e4ff3325f1ea7215eead03f300bb/espnet/nets/batch_beam_search.py#L338-L343
I found the comment.
~~@ShigekiKarita Is it OK to change `torch.arange(n_batch)` -> `torch.arange(len(runnning_hyps))`?~~
I misunderstood the problem. Let me debug more.
The problem is that `yseq.shape` is `(20, 10)` but `length` is `[20,20,20,...]`.
https://github.com/espnet/espnet/blob/67ca53d61932e4ff3325f1ea7215eead03f300bb/espnet/nets/batch_beam_search.py#L336
Maybe here should be `running_hyps.length[:] = yseq_eos.shape[1]`?

Resolving patch:
diff --git a/espnet/nets/batch_beam_search.py b/espnet/nets/batch_beam_search.py
--- a/espnet/nets/batch_beam_search.py
+++ b/espnet/nets/batch_beam_search.py
@@ -333,7 +333,7 @@ def post_process(
             )
             running_hyps.yseq.resize_as_(yseq_eos)
             running_hyps.yseq[:] = yseq_eos
-            running_hyps.length[:] = len(yseq_eos)
+            running_hyps.length[:] = yseq_eos.shape[1]
 
         # add ended hypotheses to a final list, and removed them from current hypotheses
         # (this will be a probmlem, number of hyps < beam)


Test patch: