Repository: espnet/espnet

Issue:
CSJ recipe with PyTorch backend
when using PyTorch as the backend, the model outputs only \<EOS\> token in the decoding phase, while the Chainer backend is fine (under the same hyperparameter setting). 


Discussion:
Could you share the hyper parameter setting and learning curve?
Compared to chainer, gradient vanishing is likely to occur in pytorch, especially in the case of deeper network. 
Thank you for reply!

here is the epoch-acc curve
![acc](https://user-images.githubusercontent.com/6829031/49063513-fa975400-f25a-11e8-98f8-89eb56765fa8.png)

here is the epoch-loss curve
![loss](https://user-images.githubusercontent.com/6829031/49063523-fec37180-f25a-11e8-9911-1717323edd92.png)

It looks like normal through learning curve. 

here is the hyper-parameter setting
```bash
./run.sh --ngpu 1 \
         --backend pytorch \
         --etype vggblstm \
         --subsample 1_2_2_1_1 \
         --eunits 320 \
         --elayers 4 \
         --eprojs 320 \
         --dlayers 1 \
         --dunits 300 \
         --atype location \
         --adim 320 \
         --aconv_chans 10 \
         --aconv_filts 100 \
         --mtlalpha 0.5 \
         --opt adadelta \
         --batchsize 30 \
         --maxlen_in 800 \
         --maxlen_out 150 \
         --lm_weight 0 \
         --beam_size 20 \
         --penalty 0.1 \
         --maxlenratio 0.5 \
         --minlenratio 0.1 \
         --ctc_weight 0 \
         --epochs 15 
```
Basically, The parameter setting follows the first result in the `RESULT` file.
Besides, for fast testing, I change the training data to eval1 and train the model using `Chainer` and `PyTorch` for 1 epoch, respectively. 
The model trained by `Chainer` is normal, while the model trained by `PyTorch` still the same problem.
The training seems to be no problem.
It seems that there is a bug in decoding.
Could you also attach one of the decoding log?
I tested with current master, and it seems to be fine.

```
$ head exp/train_nodup_pytorch_vggblstm_e4_subsample1_2_2_1_1_unit1024_proj320_d1_unit1024_location_adim320_aconvc10_aconvf100_mtlalpha0.5_adadelta_sampprob0.0_bs32_mli800_mlo150/decode_eval1_beam20_esnapshot.ep.11_p0.0_len0.0-0.0_ctcw0.3_rnnlm0.3_2layer_unit650_sgd_bs256/ref.trn
あ ー の ー 細 か い あ の ー 携 帯 モ バ イ ル 端 末 を 持 っ て た り と か あ る い は 彼 女 の 場 合 だ と イ ン タ ー ネ ッ ト ホ ー ム ペ ー ジ を た と 多 分 ホ ー ム ペ ー ジ を 見 な が ら 何 か 情 報 を 入 力 し て い る 質
問 と か を 入 力 し て い る と い う よ う な こ と な ん で し ょ う け ど も (A04M0123-A04M0123_0226312_0238976)
て ん (A04M0121-A04M0121_0559918_0560132)
こ ち ら の 方 は 日 本 語 女 声 ん に 注 目 し た 場 合 に 比 べ 評 価 値 が 低 く な っ て お り ま す が (A01M0110-A01M0110_0377782_0384419)
え ー と い じ ょ ん 評 価 の 結 果 分 か り ま し た (A03M0106-A03M0106_0714340_0716197)
い た だ き た い ん で す け れ ど も (A01M0137-A01M0137_0564460_0566854)
お ー し ま っ た (A04M0121-A04M0121_0010493_0011222)
良 い 近 似 が 得 ら れ る か ど う か を 分 析 に よ っ て 確 か め て み ま し た (A01M0097-A01M0097_0662303_0665667)
こ の 表 で す (A04M0121-A04M0121_0679682_0680371)
で え ー (A04M0051-A04M0051_0438134_0438799)
テ キ ス ト が 残 り ま す か ら え 以 前 の 発 話 に つ い て え え 聞 き 返 す と い う 表 現 は え ー 一 つ も 出 て き て な い と (A04M0051-A04M0051_0718240_0724997)
```

If you use old version, please test with current master.
sorry for late reply

here is the decoding log,
[decode.1.log](https://github.com/espnet/espnet/files/2628898/decode.1.log)

I currently use this commit `df13d7315daf83712a0d7b4e8cd54772f7fa9a94`, which I cloned from the GitHub just 2 weeks ago.

may be this issue related to my system environment?
Thank you for sharing!
It seems that there is a bug when calculating max & min output length.
```
2018-11-26 16:24:20,697 (e2e_asr_th:2328) INFO: max output length: 1
2018-11-26 16:24:20,697 (e2e_asr_th:2329) INFO: min output length: 0
```
I will investigate about it.

For quick fixing, please set `--batchsize 0` in `asr_recog.py`  in `run.sh` as follows:
```bash
        ${decode_cmd} JOB=1:${nj} ${expdir}/${decode_dir}/log/decode.JOB.log \
            asr_recog.py \
            --ngpu ${ngpu} \
            --backend ${backend} \
            --debugmode ${debugmode} \
            --verbose ${verbose} \
            --batchsize 0 \
            --recog-json ${feat_recog_dir}/split${nj}utt/data.JOB.json \
            --result-label ${expdir}/${decode_dir}/data.JOB.json \
            --model ${expdir}/results/${recog_model}  \
            --beam-size ${beam_size} \
            --penalty ${penalty} \
            --maxlenratio ${maxlenratio} \
            --minlenratio ${minlenratio} \
            --ctc-weight ${ctc_weight} \
            --rnnlm ${lmexpdir}/rnnlm.model.best \
            --lm-weight ${lm_weight} &
        wait

``` 

I found the reason.
https://github.com/espnet/espnet/blob/02941c89e4826fe6104afbea2cdf2e305ee293c0/espnet/nets/e2e_asr_th.py#L2323-L2327
Here we calculate LongTensor * float value.
But it will be 0 :(
```
In [4]: import torch
In [5]: lens = torch.LongTensor([324])
In [6]: max(lens) * 0.5
Out[6]: tensor(0)
In [7]: max(lens)
Out[7]: tensor(324)
```
I don't know why... This is very dangerous....
Anyway, I will fix it.

Resolving patch:
diff --git a/espnet/nets/e2e_asr_th.py b/espnet/nets/e2e_asr_th.py
--- a/espnet/nets/e2e_asr_th.py
+++ b/espnet/nets/e2e_asr_th.py
@@ -2320,7 +2320,7 @@ def recognize_beam_batch(self, h, hlens, lpz, recog_args, char_list, rnnlm=None,
         pad_bo = to_cuda(self, torch.LongTensor([i * n_bo for i in six.moves.range(batch)]).view(-1, 1))
         pad_o = to_cuda(self, torch.LongTensor([i * self.odim for i in six.moves.range(n_bb)]).view(-1, 1))
 
-        max_hlen = max(hlens)
+        max_hlen = int(max(hlens))
         if recog_args.maxlenratio == 0:
             maxlen = max_hlen
         else:


Test patch: